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Executive Summary 

Reason for Study 


The study described herein is the third in a series of cockpit weather experiments (CWE III) 
conducted by the RTI International’s Center for Aerospace Technology, and sponsored by the 
FAA and NASA. The first in the series of experiments (CWE I) investigated the effect of a 
prototype airborne weather display on pilot performance in a full mission simulator developed 
expressly for the study of new cockpit technologies in general aviation. The second in the 
series of experiments (CWE II) further investigated the effect of weather information displays 
on pilot performance. While the cockpit weather information displays implemented in these 
two studies substantially increased the pilots’ awareness of the general location of convective 
weather in their vicinity, they did not improve the decision making of the pilots using the 
display. 

The study described herein was undertaken to investigate the potential of several 
improvements to cockpit weather displays to better support the decision making of general 
aviation pilots in avoiding hazardous convective weather conditions. 

Two methods were identified that could provide information concerning convective cell 
movement: NEXRAD mosaic image looping, and the National Convective Weather Forecast 
(NCWF) product. The purpose of the experiment reported herein (CWE III) was to determine 
the effects of the use of NEXRAD looping and the use of the NCWF product on pilot workload 
and decision making. The experiment was also specifically designed to investigate the potential 
misuse of these two techniques for display of hazardous weather information as they relate to 
decision making in IFR flight, so as to support the development of recommendations to be 
provided to the FAA and to the display manufacturers for the design and use of such displays. 

Overview of Study 

The experiment was conducted with forty eight current instrument rated general aviation pilots 
who were divided into three equal groups and presented with a challenging but realistic flight 
scenario involving weather containing significant embedded convective activity. All flights 
were flown in a full-mission simulation facility in simulated instrument meteorological 
conditions. Visibility for the pilot was essentially zero from shortly after takeoff until just 
before landing. 

The control group of 16 pilots performed the flight with access to conventional sources of pre- 
flight and in-flight weather products. In addition to the conventional weather sources, the two 
treatment groups of 16 pilots each were provided with a near real-time weather display in the 
cockpit that presented text and graphical weather products to be provided by datalink service. 
Both treatment groups were equipped with a display providing NEXRAD mosaic images, 
graphic depiction of METARs, and access to text METARs. The weather information display 
used by one of the two treatment groups incorporated a NEXRAD image looping feature 
permitting the pilot to select a rapid sequence replay of the NEXRAD images received in the 
aircraft during the flight. The weather information display used by the second of the two 
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treatment groups incorporated the National Convective Weather Forecast (NCWF) product 
overlaid on the NEXRAD display. The NEXRAD image looping display provided the pilot 
with a history of storm movement from which the pilot had to estimate the future movement of 
the storm. The NCWF display provided the pilot with a depiction of direction and speed of 
storm movement, and a forecast of the predicted location of the storm in one hour. 


Key Conclusions of Study 

• Both of the cockpit weather displays used in the experiment provided a significant 
increase in awareness to the pilot of his/her situation with respect to location, 
proximity, and direction of movement of hazardous convective weather conditions. 

• Both cockpit weather displays used in the experiment provided an incomplete 
understanding of the information required to successfully deal with hazardous 
convective weather conditions, and will require substantial pilot training to permit their 
safe and effective use. 

• Overuse of both versions of the data link NEXRAD display by the pilots, at the expense 
of accessing other essential sources of information such as ATC and weather service 
providers such as FSS, AWOS, ASOS, ATIS, etc., offset the improved situation 
awareness and other benefits provided to the pilots by the weather displays to the extent 
that the decision making performance of the pilots having the weather displays was 
found to be no different statistically than that of the pilots in the control group having 
no weather display. 

• There was no significant difference in the effect of the two different types of weather 
displays on the likelihood that the pilots would over depend on their weather display in 
their navigation and decision making in dealing with hazardous weather conditions. 

• A minimum level of training, and a curriculum providing this minimum level of 
training, is clearly required to prevent misuse of any variant of the data link NEXRAD 
cockpit weather display, and to permit an acceptable pilot workload. 

• The safe and effective use of a cockpit weather display in actual instrument conditions 
requires the support of an autopilot so as to maintain an acceptable workload for most 
pilots. 


Recommendations 

Recommendations are provided for possible incorporation in the FAA Aeronautical 
Information Manual (AIM) and draft advisory circulars. These recommendations address the 
following topics: 

• The requirement that the pilot become fully proficient in determining and maintaining a 
comprehensive awareness of the age of each of the FISDL display weather information 
products so as to be able to effectively and accurately integrate this information 
(NEXRAD image time stamps, METAR text time data, etc.) with the information 
gathered from the other sources. 
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• The requirement that the pilot be fully aware that the FISDL display does not contain 
sufficient information to support navigation, and it should not be used as a replacement 
for any aspect of approved navigation procedures and equipment. 

• The mental activity required to use the FISDL display can increase the pilot’s workload 
in instrument conditions for some pilots. An autopilot can offset this workload 
increase, freeing up the mental processes to support more effective use of the display. 
The pilots in this study reported that an autopilot is essential to their effective use of the 
FISDL display. 

Additional recommendations are provided for the consideration of the FISDL display system 
manufacturers in the areas of display content and features, provision of comprehensive user 
training for display users, and emphasis to users of the need for an autopilot to permit safe and 
effective use of the display. 

Recommendations are also provided for further research, including the conduct of further 
evaluations of specific FISDL issues to support the implementation of FISDL standards, 
further development of display enhancements and concepts for information integration, and the 
development of training curriculum for weather information displays. 
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1 Introduction 

Statistics indicate that there is, on average, one fatal general aviation accident per day in 
the United States alone (AOPA, 1999 Nall Report and current NTSB accident statistics). 
While mechanical failure accounts for only 14.1 percent of the total accidents, pilot- 
related causes account for over 73 percent of the total accidents. With an overall fatality 
rate of 83.1 percent, weather related accidents were the deadliest of the pilot-caused 
fatalities. Most fatalities involving weather were the result of controlled flight into terrain 
or other objects, spatial disorientation leading to uncontrolled flight, or pilot- induced 
structural failure of the aircraft. Some accidents attributed to other causes involved 
weather as a contributing factor as in the cases of improper IFR approach accidents. 
Windshear and crosswind also caused weather-related accidents. 

While pilot training and certification regulations to minimize pilot error have been 
implemented, there have been significant advances in technology that can offer advanced 
weather displays in the cockpit via data link. This could provide a significant advance in 
aviation safety. Conventional round dial instruments accompanied by aeronautical charts, 
approach charts, and flight service station briefs represent a few of the many separate 
pieces of data that must be accessed for safe flight. The pilot is required to integrate these 
various pieces of information into an accurate mental model of the outside world. This 
information integration task often requires increases in cognitive workload, and, 
inevitably, mistakes are sometimes made. 

Advances in display system design are attempting to reduce a pilot's cognitive workload 
by doing much of the information integration behind the scenes. These designs are 
moving toward flat-panel displays with terrain, traffic, routing, and weather all overlaid 
on a single screen, thereby fostering a more intuitive mental model of “the big picture” 
for the pilot. With reduced workload involved in mentally integrating multiple elements, 
a pilot can allocate attention elsewhere, particularly to higher-level situation assessment, 
judgment and decision making tasks. Extra attention to these tasks should reduce the 
potential for error and enable pilots to make better decisions. 

However, because human performance research has lagged well behind the display 
manufacturers, many of the performance issues are yet to be determined, and the best 
way to display weather information is not yet clear. Nevertheless, weather information 
(because of its great importance in flight safety) is a prime candidate for early 
implementation in the cockpit. 

In 1999, the Federal Aviation Administration (FAA) entered into partnerships with 
industry for the development of two Flight Information Services Data Link (FISDL) 
systems with the first of these services becoming available in 2002. In addition to these 
first two industry efforts, several other vendors are nearing certification and 
implementation of various weather information display systems. The FISDL systems 
broadcast text and graphical weather information products via data link for reception and 
display in equipped aircraft. An overview of the FISDL systems is provided in Appendix 
A, Flight Information Services Description. 
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The current FAA Aeronautical Information Manual provides a description of FISDL and 
guidance for pilots in its use. See Appendix B, of the current AIM, dated February 21, 
2002. Guidance material for FISDL avionics manufacturers and service providers is 
available in the RTCA document DO-267, Minimum Aviation System Performance 
Standards (MASPS) for Flight Information Services-Broadcast (FIS-B) Data Link, dated 
March 27, 2001. 

1.1 Potential Issues with Datalinked Cockpit Weather Information Displays 

The introduction of datalinked weather information will present pilots with new 
challenges of interpretation, prediction, and action. Studies to date suggest that a “keep 
out of the red” heuristic may be adopted when, for example, viewing a NEXRAD 
baseline reflectivity product that indicates amount of rainfall according to a color-coding 
scheme (Yuchnovicz, Novacek, Burgess, Heck, & Stokes, 2001). Of course, the cessation 
of red cells (indicating areas of heavy rainfall) does not imply the cessation of peril. 

Areas of low visibility, turbulence and windshear may not appear as coded zones in 
certain weather products. Thus, pilot interpretations based on a faulty understanding of 
the weather information display, such as the red-cell avoidance heuristic, can be 
extremely dangerous. There is also evidence that some pilots interpret delayed data- 
linked weather information as though it were real time and definitive, instead of delayed 
and probabilistic. Inappropriate interpretations of cockpit weather information could have 
serious consequences for aviation safety. 

1.1.1 Use of Display to Predict Location of Hazardous Weather 

Pilots may use the cockpit weather information for prediction. One potentially significant 
issue in the use of displayed weather for prediction and forecasting is that the weather 
information is not real time, which is different from most other cockpit data including the 
data provided by on-board weather radar. In the best of circumstances, the latest 
graphical NEXRAD products will already be five to seven minutes old when received 
from the weather service provider for transmission to the aircraft and will be refreshed at 
a rate of 5-7 minutes. Thus, the data available to the pilot via the cockpit weather display 
could be as old as 14 minutes. It is not clear whether pilots will try to extrapolate, from 
delayed data, the current position of storm cells or adopt a more conservative approach. 
Also, it is not known if pilots will account for the age of the weather data in their 
extrapolations. If pilots do choose to use the weather information displays for prediction, 
their abilities to make accurate storm movement forecasts from delayed data is unknown. 

Pilots may use their interpretations of the weather information to take actions. There is 
concern, however, that some pilots might try to use the data-linked weather information 
to make tactical control actions (e.g., to weave their way between areas of perceived 
danger). The current level of technology does not afford weather information displays 
enough accuracy to be used for tactical navigation or as a substitute for traditional 
navigation aids (e.g., VORs). The current level of technology only allows pilots to use 
datalinked cockpit weather information displays safely for strategic use (e.g., to plan a 
route around possible danger zones). Thus, pilots might become overconfident in their 
ability to judge accurately where it is safe or unsafe to fly and use their interpretations of 
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the weather information to make tactical decisions and actions, which could be extremely 
hazardous. 

1.1.2 Mode Awareness 

Mode awareness concerns the extent to which operators perform actions that are 
appropriate given the particular mode of system operation. Mode errors occur when 
operators take a particular action or interpret information in ways that are appropriate in 
an inactive mode of system operation, but inappropriate in the active mode of system 
operation. Thus, mode errors are likely to occur when the operator has failed to remember 
the active mode of system operation (Norman, 1988). Breakdown in mode awareness of 
cockpit weather information displays could have large impacts on pilot workload and 
performance. Failure to appreciate the scale selected on a weather information display, 
for example, can cause a pilot to fly too close to hazardous weather conditions to permit 
safe flight. 

Mode awareness issues are particularly important in automated cockpits e.g., some 
autopilots have numerous modes of operation (Sarter & Woods, 1995). Note that cockpit 
weather information displays may have more modes of operation than most standard 
autopilots. Mode errors most often occur during automated performance or under 
conditions of high workload (Wickens & Hollands, 2000). System designers can aid 
mode awareness by making the active mode of the system salient. 

1.1.3 Cognitive Tunneling 

Pilots may overtrust cockpit weather information displays, which could influence their 
normal weather information processing strategies. For instance, pilots may over rely on 
cockpit weather information displays at the expense of failing to sample conventional 
weather information sources. Such effects are generally referred to as cognitive 
tunneling. 

There is evidence that cognitive tunneling effects are exacerbated under times of high 
workload or high stress, i.e., under attentional narrowing, (Moray & Rottenberg, 1989). 
Also, under time stress it is known that operators attend more to information sources that 
are visible, readily interpretable, and directly in their forward field of view (Wickens et 
al, 1998). For instance, Wickens and Kramer (1997) found that novice pilots tended to 
cognitively tunnel on what they considered to be the most important cockpit instrument, 
and failed to monitor other instruments that also provided important information when 
performing high-workload maneuvers. Thus, pilots who overtrust cockpit weather 
displays may fail to sample other sources of important weather information, especially in 
situations characterized by high time stress or high workload. For instance, pilots who 
cognitively tunnel on cockpit weather information displays may be less likely to 
collaborate with NAS ground weather sources (AFSS, EFAS, HIWAS, ASOS, etc.) 
concerning hazardous weather. There may be serious implications for aviation safety if 
such effects are found with cockpit weather displays. There is evidence, however, that the 
negative effects of overtrust, high workload and stress (e.g., cognitive tunneling effects) 
can be attenuated through increased experience (Stokes, Kemper, & Kite, 1997; Stokes & 
Kite, 1994; and Mandler, 1984) and training (Johnston & Cannon-Bowers, 1996). 
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1.1.4 Map Orientation 

Some cockpit weather information displays are fixed displays, in which the north 
direction is permanently aligned with the top of the display (i.e., a north-up display). In 
such displays the position of the ownship symbol moves in relation to the stationary map 
to show the aircraft’s current position. Thus, north-up displays present information about 
ownship ’s location in a world-centered reference frame, that is, in terms of the ownship ’s 
position on the globe (Aretz, 1991). 

Conversely, rotating displays are those in which ownship ’s direction of travel is 
constantly aligned with the top of the display (i.e., track-up displays). In track-up displays 
the ownship icon is stationary. To indicate change in flight path direction, the map rotates 
beneath the ownship symbol until the current direction of travel is depicted at the top of 
the display. Thus, track-up displays present information about ownship position in an 
ego-centered reference frame, that is, in a manner consistent with how the world is 
viewed through the eyes of the pilot (Aretz, 1991). 

Past research has shown that different tasks benefit from different orientations. Local 
guidance tasks, like flight control, benefit from track-up display orientation because the 
pilot does not have to perform the mental rotations necessary to make the display 
congruent with the forward field of view (Aretz, 1991; Harwood, 1989) 1 . Because track- 
up maps rotate, however, the positions of landmarks change as flight path changes. Thus, 
if it is important for the pilot to develop a global awareness (i.e., cognitive map) of the 
world or to identify their location relative to specific landmarks (e.g., restricted areas), 
then a north-up orientation is recommended (Aretz, 1991; Harwood, 1989). Also, fixed 
displays (i.e., north-up displays) facilitate better communications between other pilots in 
the airspace and air traffic controllers (Baty, 1976 as cited in Wickens et al. 1996). 

It is not clear whether the optimal orientation for cockpit weather displays is track-up or 
north-up. Currently, most weather displays have a north-up orientation, which is probably 
optimal for strategic planning and collaboration with ATC concerning hazardous 
weather. Currently, some cockpit weather information displays allow pilots to select 
between track-up and north-up orientations, which could cause confusion. Thus, map 
orientation is an important consideration for new datalinked cockpit weather information 
technologies. 


1 When navigating to landmarks within an ego-centered reference frame there is a direct correlation 
between landmarks depicted on the track-up display and the position of those same landmarks in the pilot’s 
forward field of view. That is, a landmark that is depicted on the map 15° to the left of ownship track will 
be seen 15° to the left in the pilot’s forward field of view. To navigate towards this landmark, the pilot must 
make a 15° left turn. However, the congruence between landmark locations out-the-widow and depicted on 
a north-up display are not quite as intuitive. This incongruence between out-the-window and north-up 
reference frames is exacerbated when an aircraft is flying south on a north-up display. For instance, in order 
to navigate towards a landmark that is depicted 15° to the left of the ownship icon, while traveling south on 
a north-up display, would require a 15° right turn. 
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1 .1 .5 Appropriate Use of Colors 

Color is a visual cue that is incorporated into many cockpit weather information products. 
The great advantage of color is its strong association with population stereotypes and 
logic. To capitalize on population stereotypes it is suggested that red be used to denote 
warning, yellow be used to denote caution, and green be used to denote satisfactory status 
(Stokes, Wickens, & Kite, 1990; Proctor & Van Zandt, 1994). Tullis (1981) and Kopala 
(1979) noted that pilots prefer interacting with colored displays. Stokes et al. (1990) 
reported that pilots believed the use of color increased the pictorial realism of displays. 
Color displays have also been shown to improve pilots’ abilities to detect details in 
simulated landings and response accuracy when access to necessary information is brief 
(Stokes, et al., 1990). 

Although color-coding can be a useful tool, several issues must be addressed when 
introducing color-coding into weather information displays. Because of limits of absolute 
color judgments within the sensory continuum, no more than five or six distinct colors 
should be incorporated into cockpit weather information displays. Stokes et al. (1990) 
recommends yellows, greens, oranges, and reds for displays. Also, colorblindness is an 
important issue to consider, because approximately 10% of pilots suffer from some form 
of colorblindness (Sekular & Blake, 1994) with red-green confusions being the most 
predominant. Furthermore, pilots that are red-green colorblind are perhaps at greatest risk 
because most cockpit weather information displays employ red to highlight areas of 
hazard and green to highlight areas of non-hazard. Thus, cockpit weather displays should 
incorporate redundant coding schemes so older pilots and pilots with color perception 
impairments can discern hazard/no hazard information in ways unrelated to color. 
Appropriate use of color is a major human factors’ consideration for cockpit weather 
information displays. 

1 .1 .6 Undertrust of Cockpit Weather Information Displays 

Pilots may not fully utilize cockpit weather information displays because they undertrust 
or even mistrust the technology. Trust in technology should be calibrated (i.e., directly 
proportional) to its reliability (Wickens et al., 1997). There is evidence, however, that 
human trust in technology is not always well calibrated (see Liu, Fuld, & Wickens, 

1993). Undertrust may occur for several reasons. For instance, users may not understand 
the operation of the hidden components of the technology (Wickens et al., 1997). Also, 
new technology may be confusing to operators or operators may underestimate the 
accuracy of the technology, which are also conditions that invite undertrust (Parasuraman 
& Riley, 1999;Sorkin, 1989). 

Pilot undertrust in cockpit weather information displays could be a major consideration. 
For instance, low-resolution weather displays “distort” the weather situation by making 
the weather appear more hazardous than is actually the case (Novacek et al., 2002). Thus, 
the information presented by the weather display may not match pilots’ out-the-window 
view. Such false alarms are likely to lead to mistrust and abandonment of weather 
information displays (see Parasuraman, Maoulaoua, & Hillbum, 1999). Thus, pilots who 
do not understand the limitations of cockpit weather information displays or how to 
appropriately utilize them are likely to suffer from miscalibrated levels of trust. Methods 
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to prevent and remedy undertrust include training (Norman, 1988) and improved display 
design (Landauer, 1995, Norman, 1988, Sorkin, Kantowitz, & Kantowitz, 1988; Sorkin & 
Woods, 1985). 

1 .1 .7 Usability of Cockpit Weather Information Displays 

Usability assessment measures the extent to which system design optimizes the 
interaction between the users and the system. Usability analyses often include measures 
of the system’s ease of learning, ease of use, ease of remembering, frequency of errors, 
and user subjective satisfaction (Nielson, 1993). According to Schneiderman (1992) 
usability analyses should consider three classes of end-users: Novice (users who know 
the task but have little experience with the system), Knowledgeable intermittent (users 
who know the task and understand the system, but use the system infrequently), and 
Expert frequent users (users who have intimate knowledge of the tasks and also use the 
system frequently). Cockpit weather information displays should be subjected to usability 
criteria within all three classes of users to detect information-processing bottlenecks. 

There are advantages to conducting usability assessments. For instance, systems with 
high-usability ratings have been shown to reduce the cognitive workload of the user 
(Mayhew, 1999). Usability evaluation is also cost effective, as some developers report a 
100-fold return on usability investments (Usability.gov, 2001). Usability assessments are 
necessary to optimize pilots’ interactions with cockpit weather information displays. 

1 .1 .8 Development of Decision Support Systems for Cockpit Weather 
Information Displays 

Cockpit weather information displays are a type of decision support system. Decision 
support systems are defined by Zachary (1988) as any interactive system that is designed 
to improve decision-making by extending the operators’ cognitive limitations (as cited in 
Wickens et al., 1997). The weather displays do not make decisions for pilots, but provide 
new kinds of information to pilots concerning hazardous weather. Thus, the introduction 
of these technologies into the cockpit forces pilots to make new kinds of interpretations, 
predictions, and actions. 

When pilots are flying in weather, there are a number of judgments and decisions that 
must be made: 1) pilots must determine if there is hazardous weather, 2) pilots must 
determine their proximity to the hazardous weather, 3) pilots must determine if the 
hazardous weather is going to impact their flight path, and 4) pilots must decide on a 
course of action (i.e., a revision to the flight plan) to avoid flying into the hazardous 
weather. The current cockpit weather information products were specifically designed to 
support pilots in detection of potentially hazardous weather and in determining their 
proximity to potentially hazardous weather. Some of the current displays, however, are 
not explicitly designed to support the pilots’ higher-level decision making. That is, the 
displays do not incorporate support tools to help pilots determine if hazardous weather is 
going to impact their current flight path. Nor do the current displays incorporate support 
tools to help pilots select actions (i.e., revisions to flight plan) that might be taken to 
avoid conflicts with hazardous weather. Thus, it would be helpful to pilots if high-level 
decision support tools were integrated into cockpit weather information displays. 
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There is a great deal of theory and empirical results that researchers can rely on to 
develop high-level decision support systems. In fact, such decision support tools exist in 
other domains. For instance, Schraagen (1997) described a fire-fighting decision support 
tool that was developed to help firefighters predict the spread of fire. Also, the fire- 
fighting support tool could make recommendations to firefighters regarding appropriate 
actions. Today’s cockpit weather information tools are slightly modified versions of 
meteorology weather displays. There is a limit to the extent to which pilots can utilize 
such technologies during flight to make accurate judgments concerning hazardous 
weather. When flying through weather pilots, must make complex decisions concerning 
hazardous weather under high time stress and high workload with access to delayed 
information. Even expert meteorologists, when operating under similar conditions, would 
likely find such decisions difficult. 

The upper limit of pilot processing capabilities could easily be exceeded as datalinked 
information is introduced into the cockpit. Information concerning weather avoidance, 
terrain avoidance, and conflict (with other aircraft) avoidance will soon be “pumped” into 
the cockpit. Pilots and controllers will be left to integrate the information to make high- 
level decisions. Thus, development of decision support tools to aid high-level decisions is 
crucial to the ultimate level of usefulness attained by datalinked cockpit information 
technologies (e.g., cockpit weather information displays). 

1 .2 Summary of Previous Cockpit Weather Experiments 

The study described herein is the third in a series of cockpit weather experiments (CWE 
III) conducted by the RTI International’s Center for Aerospace Technology, sponsored by 
the FAA and NASA. The first in the series of experiments (CWE I) investigated the 
effect of a prototype airborne weather display on pilot performance in a full mission 
simulator developed expressly for the study of new cockpit technologies in general 
aviation (Yuchnovicz, et al., 2001). The objective of the experiment was to investigate 
the potential for misuse of weather information, and thus provide information to the FAA 
for providing guidance to pilots and recommendations to manufacturers concerning the 
development of weather display technologies. CWE I was conducted with current 
instrument rated pilots who were presented with challenging but realistic flight scenarios 
involving weather with significant embedded convective activity. Pilots in the treatment 
group had access to conventional weather information sources and were provided a 
weather display in the cockpit that presented text (e.g., Text METAR) and graphical (e.g., 
NEXRAD radar) weather products. In contrast, the control group was given access only 
to conventional weather sources. 

The results of CWE I indicated that the treatment group did not make better decisions 
than the control group. The pilots in the treatment group found it difficult to use the 
weather information display to determine their proximity to potentially hazardous 
convective weather, to determine if hazardous weather was going to impact their flight 
path, and to determine the path of hazardous weather. Furthermore, the pilots had 
difficulty decoding the coded text METARs. The presence of the weather information 
display also impacted the pilot’s normal procedures as they relied less on traditionally 
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available weather information sources (e.g., Flight Watch). That is, the pilots cognitively 
tunneled on the cockpit weather information display. 


The second in the series of cockpit weather experiments (CWE II) further investigated 
the effect of weather information displays on pilot performance. Based on the findings of 
CWE I, certain “improvements” were made to the weather display implemented in CWE 
II. For instance, an ownship symbol was added to make it easier for pilots to determine 
their proximity to convective weather and to assess if hazardous weather was going to 
impact their flight path. CWE II also investigated the effects of image-cell resolution of 
NEXRAD graphics (4x4 km cells vs. 8x8 km cells) on pilot decision making. All other 
aspects of CWE II and I were identical. 

Several of the results of CWE II replicated the findings from CWE I. For instance, the 
pilots found it difficult to use the coded text METAR reports. They also relied too heavily 
(i.e., cognitively tunneled) on the weather display, as they failed to appropriately utilize 
conventional sources of weather information. 

The introduction of ownship symbology did not improve the decision quality of the 
pilots. It was found, however, that ownship symbology reduced perceived pilot workload 
in using the weather information display. The reduction in workload associated with 
ownship symbology was due to the reduced cognitive load required to determine the 
aircraft position in relation to the hazardous weather conditions. 

Furthermore, the age of the NEXRAD images on the weather display led to noticeable 
errors by many of the pilots in determining their proximity to convective weather and in 
determining the rate of movement of convective weather. Pilots who calculated the age of 
the NEXRAD images experienced substantially higher workload than pilots who 
assumed the information was real time or of constant age, as precise age calculations are 
cognitively taxing. Thus, it seems pilots have a difficult time extrapolating storm-cell 
movement from delayed data. It is unclear, however, whether the pilots’ problems with 
accurately forecasting storm-cell movement were due to the difficulty of determining the 
age (i.e., uncertainty) of the weather information or were simply limitations of their 
cognitive prediction processes. 

Pilots also reported numerous types of mode errors. For instance, they sometimes lost- 
track of the range setting (i.e., mode) of the display. One pilot stated that he navigated 
much closer to the storm than he intended because he thought the 25 nm scale was active 
when in fact the active mode of the display was the 10 nm scale. Such mode-awareness 
breakdowns could lead to life-death consequences in real-world operation. 

Surprisingly, the low-resolution display (i.e., less accurate information) led to better pilot 
decision making. This finding is consistent with the stimulus area effect. The stimulus 
area effect states that larger stimulus areas, represented by the NEXRAD image cell size, 
created a greater uncertainty in the exact location of the hazardous weather, which led the 
pilots to select a track farther away from the depicted weather. It is difficult to determine, 
however, if the effect would be mediated by an out-the-window view. Pilots may not use 
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or trust weather displays at lower resolution, because the information depicted on the 
display would not match their out-the-window view. 

1 .3 Purpose of Current Experiment 

Several of the pilots in the previous two experiments expressed a desire to have 
information on storm cell movement. Two methods have been identified that can be used 
to provide information concerning convective cell movement: Looping NEXRAD and the 
National Convective Weather Forecast (NCWF) product. The purpose of the current 
experiment (CWE III) is to determine the effects of the use of NEXRAD looping and the 
use of the NCWF product on pilot workload and decision making. More specifically, the 
goal of the experiment is to investigate the potential misuse of these two sources of 
hazardous weather information as they relate to decision making in IFR flight. 

Based on the findings of the previous experiments, CWE III implemented several 
improvements to the cockpit weather information display. A graphical weather depiction 
age indicator was added so pilots, at a glance, could determine the age of the weather 
information. The introduction of the age graphic should attenuate the mental effort 
necessary for pilots to calculate the age of the weather information. This prediction is 
consistent with Stone, Yates, and Parker’s (1997) suggestion that the conversion of 
arithmetic calculation or confusing language to graphical form should attenuate strain on 
working memory. Also, decoded text METAR are provided so pilots do not have to 
spend time deciphering the coded METAR reports. While professional pilots working 
with METAR reports on a regular basis would prefer the coded METAR reports because 
they substantially reduce the time required for their assimilation, many GA pilots have 
limited experience with deciphering coded METARs and the decoded reports should 
substantially decrease pilot workload. A flight-path depiction was added to make it easier 
for pilots to determine if hazardous weather would impact their flight path. A 25 nm 
range -ring was added to aid the pilots in the determination of their proximity to 
hazardous weather (when using the 10 mile and 25 mile scale displays). Also, the 25 nm 
range -ring is invariant across display scale selection (i.e., range modes), which should 
attenuate mode errors associated with the display scale selection. 

1.4 Survey of Relevant Literature 

Pertaining to the display of data-linked weather information, relatively little documented 
research has been conducted to date. The next generation of research must begin in order 
to catch up to rapidly emerging technology. 

Past studies have primarily focused on situational awareness (Hansman, & Wanke, 1989; 
and Lee, 1990), and expert/novice strategic decision making, (mostly making go/no go 
decisions), (Driskill, Weissmuller, Quebe, Hand, Dittmar, Metrica, & Hunter, 1997; 
Dershowitz, Lind, Chandra, & Bussolari, 1996; Fisher, Brown, Wunschel, & Stickle, 
1989; Wiggins, Connan, & Morris, 1995; and Wiggins & O'Hare, 1995). Little has been 
done to examine the possible “tactical” decisions made during flight, and none have 
looked at this issue in a full-mission simulator. 
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One of these issues is the impact of textual versus graphical presentation of weather 
information on pilot decision making. A particularly relevant study was a comparison of 
textual presentation versus graphical presentation of weather information undertaken at 
the Lincoln Laboratory of MIT (Lind, et. al., 1994) that provided a valuable first step by 
looking at the influence of data-link provided graphical weather on pilot decision making. 
When compared to strictly text information, the graphical information caused pilots to 
become more confident in their assessment of the weather, and to make better Go/No Go 
decisions as well as flight path change decisions. Although very valuable, this study was 
performed in an office setting without a high-fidelity flight simulator and, therefore, 
without factors that come into play in an operational setting. The study suffered from 
low-cognitive fidelity as decisions were made based on static images presented at 
selected points during a scripted scenario. 

Spatial displays have also been found to improve accuracy over text in presenting 
information for an analog operation/tactical decision task (Wickens & Scott, 1983). 

All these findings are consistent with the multiple resource theory of attention and the 
proximity/compatibility principle (Wickens, Gorden, & Liu, 1998). These findings 
suggest that if an individual is to perform a visual-spatial task (such as navigating an 
aircraft through the airspace), then the information needed to perform that task should be 
presented in a visual- spatial way (e.g. as graphics, rather than, for example, a visual- 
verbal way such as in teletyped weather products). 

The studies performed to date represent a fraction of the studies that are needed with the 
introduction of new technologies to resolve issues that arise in implementation and 
operational use. 
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2 Participants 


This experiment was a cooperative effort between the Federal Aviation Administration 
(FAA), National Aeronautics and Space Administration (NASA) and RTI International 

2.1 FAA Data Link Office 

The FAA Flight Information Data Link Office (AUA-460) was a prime sponsor for this 
experiment. This effort was undertaken to support the development of guidance for the 
use of cockpit weather displays in the National Airspace System. 

2.2 NASA AWIN Project 

The NASA AWIN (Aviation Weather Information) project was also a prime sponsor, and 
provided technical support and contract management for this experiment in partnership 
with the FAA. The AWIN effort is an element of the Weather Accident Prevention 
Project of NASA’s Aviation Safety Program. 

2.3 RTI International 

The experiment was designed and conducted by the Flight Systems Engineering Office of 
RTI International, located in the Hampton, Virginia, office. Mr. James Murray, a retired 
FAA air traffic controller, National Airspace System automation specialist, air traffic 
manager, and consultant to RTI International in many previous research efforts, provided 
air traffic control expertise in the design and execution of the experiment. Mr. Rickey 
Thomas of Kansas State University, also a consultant to RTI International, provided 
expertise in preparation for and analyses of the data. Consultants from Booz- Allen Inc. 
assisted in the conduct of the experiment. 
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3 Methodology 


The objective of this experiment was to determine the effects of the use of NEXRAD 
image looping and the use of the NCWF product on pilot workload and decision making. 
More specifically, the experiment investigated the potential misuse of these two sources 
of hazardous weather information as they relate to decision making in flight in instrument 
meteorological conditions. 

3.1 Experimental Hypotheses 

The experiment was designed to answer several experimental hypotheses. The first two 
hypotheses concern the impact of NEXRAD looping on decision making and workload. 

1) Pilots using a NEXRAD image looping display will make significantly better 
decisions with respect to convective weather than pilots without a weather 
information display. 

2) Pilots using a NEXRAD image looping display will experience significantly 
higher workload than pilots without a weather information display. 

The second two hypotheses concern the impact of the NCWF product on decision making 
and workload. 

3) Pilots using the NCWF product display will make significantly better decisions 
with respect to convective weather than pilots without a weather information 
display. 

4) Pilots using the NCWF product display will experience significantly higher 
workload than pilots without a weather information display. 

It is expected that with sufficient training, careful use of the cockpit weather information 
display, and prudent pilot procedures in instrument flight conditions, the emerging 
cockpit weather display products will provide substantial improvements to the safety of 
flight. It is important to note that this experiment, like the previous experiments, was 
designed specifically to identify potential hazards in the design and use of cockpit 
weather displays. 


3.2 Experiment Design 

Every aspect of the design of this experiment was undertaken with the objectives of the 
experiment in mind, including participant pilot selection, participant pilot training, and 
the mission scenario. Pilots were selected so as to provide a wide and representative 
range of the experience and knowledge of the population of general aviation pilots who 
might use these emerging cockpit weather display products. The training provided to the 
participant pilots was tailored to provide them with sufficient familiarity with the 
experimental equipment to successfully accomplish the mission scenario, while at the 
same time creating a reasonable probability that within the sample of pilots, potential 
hazards in the use of the equipment might become apparent. Likewise, the mission 
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scenario incorporated in the experiment was selected to ensure that it could be 
accomplished by the average pilot with careful attention to the instrument flight 
procedures, but offered sufficient opportunity for observation of human error in the use of 
the cockpit weather displays where such hazards might exist. 

The experiment was designed to have certain desirable properties. It was moderate in 
length (approximately one hour depending on pilot actions) in order to eliminate fatigue- 
related effects. It was made up of sufficiently independent phases to test responses to 
discrete weather conditions. The incident density was to be plausible and would be 
designed to occur while crossing informational boundaries (where most decision-related 
errors are likely to occur). The mission scenario and cockpit simulator were to be 
sufficiently realistic such that the participant pilot would be immersed in the experiment. 

The experiment employed a three (Display Type) by two (Decision Type) mixed design. 
The between factor was Display Type whereby pilots were randomly assigned into one of 
three groups (i.e., No Display vs. Looping NEXRAD vs. NCWF). Performance 
differences between the three groups could then be attributed to differences between the 
control (no display) and treatment conditions (NCWF and Looping NEXRAD). The 
within- subjects factor was Decision Type (Temporal vs. Spatial). All participants made 
both types of decisions. 

3.3 Participant Pilot Sampling Procedure 

Participant pilots were recruited for the experiment through advertisement via the Internet 
and through local flying organizations. A convenience sample of 48 current instrument 
flight rated pilots was selected for participation in the study. An additional 6 IFR and 
current pilots were recruited as back-up participants. 

3.3.1 Random Assignment of Participant Pilots to Experimental Conditions 

Forty-eight pilots were randomly assigned to the experimental conditions (i.e., NCWF, 
Fooping NEXRAD, and Control). Random assignment of pilots to experimental 
conditions minimized the probability that pilot characteristics (e.g., total flight hours) and 
uncontrolled experimental characteristics (e.g., time of day) would systematically differ 
across conditions. Thus, any differences in outcome across conditions are likely due to 
type of display rather than differences between pilots. 

The random assignment procedure was restricted to ensure an equal number of pilots 
were selected into each condition (n = 16). The procedure required that any given 
condition could not be run until all the other conditions were represented once (i.e., 
sampling without replacement). For example, the first three pilots constituted a 
randomized block where the first pilot was randomly assigned to one of the three 
experimental conditions. The second pilot was randomly assigned to one of the two 
remaining experimental conditions, and the third pilot was assigned to the one remaining 
experimental condition. This assignment completed a block of randomized conditions. 

The fourth pilot started the second randomized block, and was randomly assigned to one 
of the three conditions. The fifth pilot was randomly assigned into one of the two 
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remaining conditions, and the sixth pilot was assigned to the one remaining condition; 
and so on. 

Maintaining an equal sample size across conditions had several advantages. For instance, 
equal sample size ensured that each display condition contributed equally to the statistical 
analyses, consistent with the logic of experimental design (Keppel, 1991). Moreover, 
equal sample size allowed for more robust statistical tests and increased statistical power 
(Glass, Peckham, & Sanders, 1972; Milligan, Wong, & Thompson, 1987; and Keppel, 
1991). 

3.4 Experiment Apparatus 

The experiment was performed in a full-mission flight simulator to provide a realistic 
operational environment. Three major components comprised the experimental system: 
pre-flight planning tools, the flight simulation facility, and the weather information 
display. 

3.4.1 Pre-Flight Planning 

Each pilot was given 30 minutes to plan the flight. The following flight planning tools 
were provided: 

• A written transcript of a telephone Flight Service Station (FSS) weather briefing 
(provided in Appendix A, Preflight Weather Briefing) 

• Aircraft Flight Manual 

• Aeronautical charts (sectional and IFR low-altitude enroute) 

• Blank flight logs 

• Partially completed flight plan forms (each pilot given same route). 

3.4.2 Flight Simulation Facility 

The flight simulation facility consisted of a full-mission simulator that provided a 
simulation of a complex, high-performance single-engine, single-pilot IFR-equipped 
airplane having the major features and performance of a Piper Malibu PA-46-31 OP. The 
instrument panel, however, was substantially simplified from that of the Piper Malibu to 
that of a typical low-end general aviation aircraft to permit a wide population of general 
aviation pilots, not all who would be familiar with the more sophisticated Piper Malibu 
instrumentation, to participate in the experiment. The key elements of the simulation 
facility are illustrated below in Figures 3.4.2-1 and 3.4.2-2. 
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Figure 3.4.2-1. Cockpit Simulation Facility (CRF) Diagram 



Figure 3.4.2-2 Cockpit Simulation Facility Instrument Panel and Controls 
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This full-mission simulator facility consisted of three major sections as follows: 

• Aircraft Cockpit Simulator - Consisted of the cockpit mockup with controls, 
instruments, radios and indicators. The flight control system incorporated a 
mechanical simulation of control forces in pitch, roll and yaw. A closed-circuit 
television camera was mounted behind and above the pilot’s left shoulder to provide 
live images from the cockpit to the Scenario Controller and Observer positions. The 
simulated cockpit instrumentation is shown below in Figure 3 A. 2-3. The refresh rate 
of the cockpit instrumentation displays was 30 Hertz. The weather information 
display was located between the primary and secondary instruments to maximize its 
visibility and probability of use. 

• Simulation Facility and Scenario Controller and Observer Positions - Consisted of 
the master control station used for scenario generation and for selection, monitoring 
and recording of flight progress. It provided the operator with displays of all control 
positions, radio and instrument switch positions, instrument displays and the Out-the- 
Window scene (as presented to the subject pilot). A weather data display consisting of 
NEXRAD images was provided for the scenario controller, and enabled the observers 
to track the flight’s progress relative to the weather. A video image of the cockpit 
from the camera was provided to enable the observers to monitor the participant 
pilot’s actions. Live audio of all radio transmissions between the pilot and the Air 
Traffic Controller, Flight Watch, ATIS, etc., was available to the simulation scenario 
controller and to the observers. An intercom audio network was provided which 
permitted private conversations between the scenario controller, observers, and air 
traffic controller positions. The ability for the pilot and air traffic controller to 
communicate was also provided by the same intercom system. All intercom traffic 
was recorded on the audio track of the video recording. 

• ATC Controller Position - Consisted of a custom ATC workstation developed for 
experiments of this type and a weather display that provided the latest NEXRAD 
images enabling the ATC controller to track the flight’s progress relative to the 
weather. The NEXRAD images were updated every minute. A display of the current 
pilot-selected communication frequencies was also provided so that the ATC 
controller could verify that the pilot was contacting ATC on the correct frequency 
before responding to an initial contact. 
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Figure 3.4.2-3. Cockpit Simulation Facility Instrument Panel and Controls 

Additional detail regarding the experimental system is provided in Appendix B, Cockpit 
Research Facility Description. 

3.4.3 Cockpit Weather Information Display 

The weather information display system used in this experiment consisted of two key 
components, a PC-based computer and a prototype display unit originally developed by 
NavRadio Inc, which was subsequently acquired by Honeywell Inc. 

The computer was a PC-based workstation running Microsoft Windows NT and custom 
software. The PC was used to record, process and playback the NEXRAD/METAR data 
gained through a C-band satellite downlink receiver. Using custom software, the PC 
sequenced through the database of previously recorded NEXRAD/METAR information 
and displayed the images on the weather display. The PC also contained the software to 
depict the moving map, mode/scroll control, airport/navaid information and map 
database. 

The display unit contained a flat-panel LCD display of 320 x 240 resolution, a joystick 
and mode-select softkeys. The display unit provided only the display and control function 
of the system. The computational effort resided in the PC workstation. 
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The weather display enabled the pilot to select weather information and included the 
following features: 

• NEXRAD image depiction 

• Coded METAR text reports 

• Decoded METAR text reports 

• Depiction of graphic METAR symbology 

• Graphic symbology of airports, VORs, major highways and state boundaries 

• Zoom and scroll capability 

• Map range scale 

• 25 nm range ring 

• NEXRAD image timestamp 

• NEXRAD image age indicator 

• Flight-path depiction. 

The key features of the cockpit weather display are depicted in Figure 3.4. 3-1 The 
placement and layout of the weather display controls are depicted in Figure 3.4. 3-2 
Details of the FISDL system, of which this display is an early prototype, are provided in 
Appendix C, Flight Information Services Description. 
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Figure 3.4.3-1. Cockpit Weather Display Features 
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Display Mode: 
Graphics 
METAR 
NEX/METAR 


Movement Mode: 
Zoom 
Scroll 
Crosshair 



Figure 3.4.3-2. Cockpit Weather Display Controls 


Two versions of the weather display were developed for the CWE-III experiment — a 
version incorporating NEXRAD image looping and a version incorporating the NCWF 
product. One of the treatment groups was given the ability to animate previous NEXRAD 
images (i.e., see a history of storm-cell movement) with a dedicated “LOOP” softkey 
selection. The second treatment group was presented with the National Convective 
Weather Forecast (NCWF) product that depicted a 1-hour forecast of storm-cell 
movement. 
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3.4.4 NEXRAD Image Looping Display 

The NEXRAD image employed a three-color palette to depict the precipitation returns 
for a given area. The arrangement of the NEXRAD precipitation intensity levels used in 
the experiment is shown in Figure 3.4.4-1. The images were updated every seven 
minutes. The pilot was able to zoom and scroll around the image to the desired view. The 
16 small-cell (i.e., 4x4 km) NEXRAD images used in the experiment are duplicated in 
Appendix D, NEXRAD Images Used in CWE III Experiment. 



NEXRAD Intensity Levels: 
. Green □ Light 

Yellow ^ 
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Severe 


Figure 3.4.4-1. NEXRAD Mosaic Image Precipitation Intensity Key 

The NEXRAD weather display employed an additional “LOOP” softkey that enabled the 
pilot, when the softkey was actuated, to display a replay of a series of up to seven 
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NEXRAD images (approximately 49 minutes) in a single cycle. Successive actuation of 
the “LOOP” softkey replayed the loop a single time for each actuation of the softkey. 

The “LOOP” softkey resided in the third softkey position on the weather display and was 
only available when a NEXRAD image was currently being displayed. The position of 
the looping softkey is normally reserved for actuation of additional information when the 
crosshair mode is selected by the first softkey. Therefore, whenever the crosshair mode 
was selected, the third softkey provided the Text/Airports/Navaids selection. Thus, the 
looping feature, and softkey annunciation, was only available when a NEXRAD image 
was currently displayed and the crosshair mode was not selected. 

3.4.5 NCWF Product Display 

The National Convective Weather Forecast (NCWF) product, developed by NCAR under 
the FAA Aviation Weather Research Program (AWRP) and implemented operationally 
by the National Weather Service Aviation Weather Center (AWC), provides current 
convective hazards and one-hour extrapolation forecasts of thunderstorm hazard 
locations. The NCWF product is currently available from the Aviation Digital Data 
Service (ADDS), through the Internet for flight planning purposes. While at least one 
weather display manufacturer is apparently studying the possibility of including the 
NCWF product in their cockpit weather display, no company has yet sought approval 
from the FAA to do so. 

The information currently provided in the FAA Aeronautical Information Manual (AIM) 
about the NCWF product as of February 21, 2002 is provided in Appendix E. 

The hazard field and forecasts in the NCWF product are updated every 5 minutes. The 
diagnostic analysis combines national radar and echo top mosaics, and cloud-to-ground 
lightning. 

In the experiment the NCWF images were updated every seven minutes and the 
Convective Hazard Detection field was depicted using a three level intensity scale as 
indicated in Figure 3.4. 5-1. The three levels corresponded to VIP levels as indicated in 
Table 3.4. 5-1. The one-hour forecast locations of significant convection (NCWF hazard 
scale levels of 3 or greater) were depicted by cyan polygons. The direction of movement 
and speed of the convective cells were indicated by vectors. Also, the altitude of the 
storm tops and storm speed (nautical miles per hour) were presented in digital symbology 
adjacent to the movement vectors. The display of the NCWF storm movement vectors, 
one-hour forecast, storm speed, and cloud tops was available in the 10 and 25 mile range 
scales. When a scale of greater than 25 miles was selected, these NCWF features were 
not displayed to reduce clutter. The NCWF images used in the experiment are provided 
in Appendix F. 
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Figure 3.4.5-1. NCWF Product Display Convective Hazard Intensity Levels 
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Table 3.4.5-1 NCWF Display Relationships to Other Sources 


Hazard Field 

Approximate Correlations 


VIL 

(kg/m 2 ) 

Ltg. 

Rate (/10 
min) 

Effect 

Airborne 

Radar 

Reflectivity 

(dBZ) 

VIP 


12 + 

15 + 

Severe Turbulence, 
Lightning Hail likely 

Level 4 Very 
Hazardous 

50 + 

5-6 

3.5-12.0 

3 - 15 

Possible Severe 
Turbulence, Lightning 

Level 3 
Hazardous 

40-49 

3-4 


0-3.5 

NA 

Light to Mod. Turbulence, 
Possible Lightning 

Level 1-2 
weak-moderate 

0-39 

1-2 


3.4.6 Graphical METAR Reports 

The graphical METARs (common to both the NEXRAD Image Looping display and the 
NCWF product display) were small graphic icons that depicted the ceiling and visibility 
for the reporting station. The coding of the graphic METARs is depicted in 
Figure 3.4.6- 1. 



Figure 3.4.6-1 Graphic METAR Key 
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3.4.7 Text METAR Reports 

An example of a decoded text METAR (common to both the NEXRAD Image Looping 
display and the NCWF product display) is given in Figure 3.4. 7-1. The pilots were 
provided decoded text METARs instead of the traditional coded text METARs. The 
English translation of the text METAR was consistent with the ICAO code. By selecting 
either the METAR function or the NEXRAD/METAR function on the Looping 
NEXRAD display or the NCWF display, then selecting the crosshair function of the 
joystick, the subject pilot equipped with one of these displays could place the crosshairs 
over any METAR graphic icon and call up on the display the text description of the entire 
METAR report for that reporting station. 



Figure 3.4.7-1 Example of Decoded METAR Text 


3.4.8 NEXRAD Image Age Graphic 

In the lower left comer of the display, a graphical thermometer-type icon displayed the 
difference between the mission system time and valid time of the weather image 
(common to both the NEXRAD Image Looping display and the NCWF product display). 
The range of the thermometer scale started at zero delay on the left side, increasing to 20 
minutes on the extreme right side, in five-minute increments. The pointer was depicted in 
yellow, and red filled the space between the pointer and the left edge of the thermometer 
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graphic. See Figure 3.4.8- 1. The general intent of the age graphic was to allow the pilot to 
discern the age of the weather information at a glance (i.e., the more red displayed the 
older the information). 

I I III 



Figure 3.4.8-1. NEXRAD Image Age Graphic 


3.4.9 NEXRAD Image Range Ring 


The range-ring depiction (common to both the NEXRAD Image Looping display and the 
NCWF product display) is illustrated in Figure 3.4. 9-1. A white 25 nautical mile ring was 
depicted around the aircraft ownship icon. The range ring was depicted when either the 
10 or 25 mile map scales were selected. Superimposed over the line was a digital numeric 
icon of “25.” 
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Figure 3.4.9-1. NEXRAD Image Range Ring and Flight Path Depiction 

The route depiction feature (common to both the NEXRAD Image Looping display and 
the NCWF product display) is also illustrated in Figure 3.4.9- 1. The route depiction 
consisted of a series of superimposed waypoints connected by a magenta line. Collocated 
next to the waypoint icon was the station identifier. The pilot was not able to declutter the 
course lines or waypoint icons, nor was the pilot able to edit the flight plan. The display 
of the courseline, waypoints and identifiers was available in the following map ranges: 

10, 25 and 50 mile scales. When a scale of greater than 50 miles was selected, the 
waypoint identifiers were not displayed. 
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4 Experiment Procedure 


The experiment procedure consisted of the following five key phases: 

1 . Experiment briefing 

2. Simulator familiarization 

3. Pre-flight planning 

4. Simulator mission 

5. Post-mission briefing 

4.1 Experiment Briefing 

The participant pilots were given a briefing of the mission objective, mission scenario, 
and an extensive overview of the simulator controls. The pilots in the treatment groups 
were also given a comprehensive overview of the operation of the appropriate weather 
display (NEXRAD Image Looping or NCWF weather forecast). The briefing scripts are 
provided in Appendix G, Experiment Briefing. 

4.2 Simulator Familiarization 

The participant pilots were provided with a familiarization session and practice flight in 
the simulator. Systems, controls and displays were explained and demonstrated. The 
simulator instructor answered any questions that the pilot had with respect to the 
operation of the simulator. Additionally, the pilots were given a hands-on training session 
on the use of the in-flight weather display system and the autopilot. 

The training was provided in an interactive environment that provided a thorough 
understanding of the equipment and its capabilities. To assure equal treatment to all 
participant pilots, the training session was heavily scripted and the pilots were trained to a 
predetermined performance level derived from the FAA Practical Test Standards for 
Instrument Pilots. 

The simulator training took approximately 90 minutes and included a practice flight with 
at least two approaches (one YFR approach and one IFR approach). Some pilots required 
as many as three IFR approaches before they met the proficiency criteria. The training 
outline and proficiency criteria are provided in Appendix H, Simulator Training and 
Proficiency Criteria. 

4.3 Pre-Flight Planning 

Each pilot was given 30 minutes to plan the mission. Weather reports and flight planning 
materials were provided. Additionally, a partially completed flight plan form was 
provided that had the route and aircraft-specific particulars completed. The pilots were 
told that the route given to them was already filed with Flight Service, but that they were 
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free to request a change of route upon the initial call to clearance delivery. The flight plan 
route is illustrated in Figure 4.3-1. They were also briefed that all the normal weather 
information services typically available in the National Airspace System (NAS) were also 
available in the simulator, including Flight Service, Flight Watch, ATIS, ASOS and 
ATC. 



4.4 Simulator Mission 

The pilots were left alone in the simulator for the mission and observed remotely. The 
mission lasted approximately one hour, depending on the pilot and route selected around 
the hazardous weather conditions. 

A team of four individuals, including a Simulator Operator, an Air Traffic Controller, an 
Observer/Test Director and a second observer conducted the experiment. The Observers’ 
primary role was to collect and record data on the Observer form (Appendix I), including 
the comments of the other members of the research team. Additionally, the Air Traffic 
Controller provided a record of his observations that were included in each participant 
pilot’s data folder. 
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4.5 Post-Mission De-briefing 

Upon completion of the mission, each pilot was given a questionnaire while still seated in 
the simulator, thus providing important subjective comments while still in the mission 
“mode.” The separate questionnaires for the control and treatment groups are included in 
Appendix J, Immediate Reactions Questionnaires. 

After completing the questionnaire, the pilots were interviewed by the experiment 
observers using the Structured Interview Guide (see Appendix K) to confirm behavioral 
actions and decisions. 

As a final step, the pilots in the treatment conditions were asked open-ended questions 
concerning the cockpit weather information displays. The questionnaire is included in 
Appendix L, Weather Display Questionnaire. 

4.6 Mission Scenario 

The mission scenario consisted of a flight to a wedding rehearsal, wedding, and dinner at 
Wallops Island. Wallops Island is located on the eastern shore of Virginia. The 
participant pilots were told that they were one of three pilots who have been close friends 
for many years since their military flying tours. The participant pilot was to assume the 
role of the friend who lives in Newport News, VA, and would be flying the aircraft to 
Wallops Island for the wedding of one of the other friends. The participant pilot ( the 
friend flying the aircraft from Newport News - Williamsburg Airport) had agreed to stop 
at the Richmond Airport enroute to Wallops Island to pick up the third friend who had 
agreed to be the best man at the wedding. 

The two friends who were traveling had planned their flight so as to leave around mid- 
day on the day of the wedding to attend the wedding rehearsal, wedding and dinner party, 
all of which were scheduled for the same evening. The rehearsal started at 1700 hours 
local. The pilot was delayed at work due to an important meeting, however, and the 
planned departure time from the Newport News airport was slipped to 1500 hours local. 
At that point, while there was more than sufficient time to fly the agreed route to Wallops 
Island and make it to the wedding rehearsal via the Richmond Airport, there was not 
sufficient time to drive from Newport News to Wallops so as to participate in the 
rehearsal or the wedding. Nor was there time for the pilot’s friend (the best man) to drive 
from Richmond in time to participate in the wedding rehearsal or wedding at Wallops 
Island. 

The flight originated at the Newport News Virginia airport. The pilot was instructed to 
fly to Richmond Virginia and pick-up the other friend (best man) on the way to Wallops 
Island. Total flight time for the two legs of the flight was about one hour. The pilots were 
briefed to assume a time of about 15 minutes at Richmond to pick up the passenger, and 
to assume that transportation would be provided from the Wallops Island Airport to the 
wedding rehearsal, which would take no more than five minutes. 

In the course of the preflight briefing, the pilot found that there was a weather front 
moving in the direction of Richmond, but that the forecast for the area would permit the 
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pilot to land at the Richmond airport to pick-up his friend. The forecast weather for much 
of the flight placed the aircraft in instrument meteorological conditions (clouds and 
occasional light rain), but the weather along the eastern shore of Virginia and at Wallops 
Island airport was forecast to be well above minimums. 

All flights were flown in a full-mission simulation facility in simulated instrument 
meteorological conditions. The pilots were in instrument meteorological conditions with 
essentially zero visibility from shortly after takeoff until reaching visual meteorological 
conditions about 15 minutes before landing. The pilots were to conduct the flight in 
accordance with all appropriate ATC procedures in conjunction with an Air Traffic 
Controller (ATC), located in an adjoining room. The ATC workstation fulfilled the roles 
of clearance controller, ground controller, tower controller, approach/departure controller 
and FSS briefer as required throughout all phases of the flight. The scripts are provided in 
Appendix M, Air Traffic Control Scripts. 

The pilot was able to access the normal in-flight weather services through VHF radio, 
including: 

• FSS - Flight Service Station 

• ATC - Air Traffic Control (tower, departure and approach) 

• FW - Flight Watch 

• ATIS - Automatic Terminal Information Service 

• ASOS - Automated Surface Observation System 

The ATC workstation presented the Air Traffic Controller with a readout of the 
frequency that the participant pilot selected on the simulator communication radio. When 
the pilot tuned the communication radio to a frequency that corresponded to a recorded 
weather message (ATIS, etc.), a prerecorded report was played through the intercom. The 
ATIS/ASOS recorded scripts can be found in Appendix N, Enroute Weather Report 
Scripts. 

If the pilot called either a Flight Service Station or Flight Watch briefer, the Air Traffic 
Controller read a scripted weather report to the pilot depending on the time of the call. 
These weather scripts can also be found in Appendix N, Enroute Weather Report Scripts. 

Actual weather data was used to assure the realism of the operational scenario. All 
weather information used in this experiment was recorded from actual weather conditions 
that existed in the geographical area of the experiment on the evening of March 27, 2000. 
The NEXRAD images were recorded during passage of multiple weather fronts through 
southeastern Virginia from a prototype satellite data gathering system provided to RTI 
International by Honeywell, Inc. All NEXRAD mosaic images used in the experiment 
were recorded with a cell resolution of 4 km. NCAR was provided the NEXRAD images 
from the evening of March 27, 2000. An NCWF NEXRAD and NCWF forecast was then 
generated for each NEXRAD image by NCAR for use in the experiment. The NEXRAD 
images and NCWF images were replayed on the weather display in the simulation facility 
cockpit. The NEXRAD images used in the experiment are provided in Appendix D, 
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NEXRAD Images. The corresponding NCWF images are provided in Appendix F, 
NCWF Images. 

To realistically reproduce actual data-linked weather products, the participant pilot 
received the NEXRAD mosaic images and NCWF weather forecast images delayed by 
seven minutes. The pilot’s weather display of NEXRAD images and NCWF images were 
initially seven minutes old, aging to 14 minutes old before receipt of the next update (of a 
seven-minute-old image). The pilot also had access to graphical and textual Aviation 
Routine Weather Report (METAR) information. The text METARs were available only 
in a decoded format. 

The NEXRAD weather display used by the Air Traffic Controller emulated the weather 
radar information typically provided at the various controller work stations. The Air 
Traffic Controller received a real-time NEXRAD image that was no more that one 
minute old. 

All the other weather data products needed to develop preflight and inflight weather 
reports for the experiment scenario were collected from the appropriate FAA sources for 
the same location, date and time captured in the NEXRAD mosaic images. 

4.7 Flight Procedures 

The mission flown by the subject pilots consisted of two flight legs, the Richmond leg 
and Wallops Island leg. 

4.7.1 First Leg of Flight - Newport News to Richmond 

The Newport News - Richmond leg of the experiment was designed primarily to 
determine the pilot’s judgment relative to the temporal issues in the use of the weather 
information displays. During the course of the first leg of the flight, between Newport 
News and Richmond, the ceiling and visibility at the Richmond airport had descended to 
below minimums (200 feet/ 1/2 mile) sooner than forecasted. Additionally, there was a 
thunderstorm approaching the Richmond airport. The only way the pilot could learn of 
these deteriorating conditions was to obtain an in-flight update of the weather. The pilots 
could gather these updates either through the weather display or by radio. 

Before reaching the initial approach fix for the Richmond airport, the weather display 
depicted a thunderstorm cell several miles to the west of the airport but headed toward 
the airport. A typical image of the weather conditions at that time as seen on the cockpit 
weather display equipped with the NEXRAD images alone is depicted in Figure 4.7-1. A 
typical image of the weather presented on the display equipped with the NCWF product 
is depicted in Figure 4.7-2. The image on the pilot’s weather display was a minimum of 
seven minutes old and could have aged up to as much as 14 minutes old. By the time the 
pilot began the approach, the actual weather cell had intensified and moved closer to the 
airport. [The ATC workstation weather display showed the storm to be approximately 
two miles northwest of the airport.] 
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Figure 4.7-1 Typical NEXRAD Image 
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Figure 4.7-2. Typical NCWF Image 

There were several possible responses to this scenario. The pilot could continue the 
approach with old data and proceed into the thunderstorm (poor decision), or, the pilot 
could decide to abandon the approach into Richmond and proceed directly to Wallops 
Island (good decision). A third option was for the pilot to ask ATC to provide a hold until 
updated weather information could be obtained and sorted-out before deciding to 
continue into the Richmond airport or proceed to Wallops Island (good or poor decision 
depending on proximity of flight path to thunderstorm). 

As the aircraft traversed the various precipitation zones — as depicted on the simulator 
operator’s NEXRAD image display — the simulator operator introduced levels of 
turbulence appropriate to the precipitation level. For flight in clear air, turbulence was not 
encountered, but when the aircraft traversed into an area depicting precipitation, a 
turbulence model was applied to the simulation and the turbulence was increased in 
proportion to the intensity level. 

If the pilot gathered weather information (either via voice or from the weather display) 
during the leg between Newport News and Richmond, the pilot was apprised of the 
rapidly changing weather and had to make a decision to either divert to Wallops Island or 
continue the approach into Richmond. This is the decision that the experiment was 
designed to uncover along with the basis for the participant pilots’ decisions. 
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If the pilot proceeded with the approach into Richmond, typical and consistent weather 
warnings were given to the pilot by ATC, including an ATIS report indicating the 
Richmond airport weather conditions had deteriorated to 200 feet/3/4 mile with thunder- 
storms in the vicinity, and a windshear warning when the pilot contacted the tower. If the 
pilot inquired of ATC in any way about the weather conditions, the weather conditions 
along the route to Richmond were described and eventually a PIREP was relayed that 
was provided by a Cessna 210 about ten minutes ahead who had experienced severe 
windshear and turbulence during the approach into the Richmond airport. To expedite the 
simulator mission, if the pilot decided to proceed with the approach into Richmond, ATC 
informed the pilot (when crossing the final approach fix) that the Richmond airport 
manager had closed the airport due to windshear and heavy lightning activity. This 
methodology would preserve the timing essential to maintaining a consistent relationship 
between the aircraft position and weather movement for all the test flights. Therefore, all 
the pilots either broke off the attempt to land at Richmond at various distances from 
Richmond, or were waved-off just after passing the final approach fix. 

4.7.2 Second Leg of Flight— Richmond to Wallops Island 

During the leg between Richmond and Wallops Island, a line of storm cells materialized 
across the direct route to Wallops Island, with one storm cell to the north of the direct 
course and one to the south. The location of this convective activity can be seen in Figure 
4.7-3 in a typical NEXRAD Image and in Figure 4.7-4 in a typical NCWF image. 



Figure 4.7-3 Typical NEXRAD Image Used In Wallops Leg Decision 
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Figure 4.7-4. Typical NCWF Image Used in Wallops Leg Decision 

The distance between the red cells was approximately 10-12 miles. The gap between the 
storms was tempting enough to suggest a corridor might exist between the areas of 
hazardous weather. These cells did not move substantially with succeeding weather 
information image updates, but slightly changed shape and size. 

The METAR graphical and textual depictions showed that the eastern shore of Virginia 
and the Wallops Island airport were well above minimums, therefore giving the pilot an 
incentive to proceed with the flight to Wallops Island. 

The pilot was monitored as to the decision to proceed between the storm cells, or 
circumvent the area of thunderstorms altogether. This part of the experiment was 
designed primarily to determine the pilot’s judgment relating to spatial interpretation 
issues in the use of the weather information display. 
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5 Results 


5.1 Pilot Characteristics 

A convenience sample of 48 pilots participated in the experiment. Of these 48 pilots, 16 
flew with the NEXRAD image looping display, 16 flew with the NCWF product display, 
and 16 flew with access to only conventional weather information sources (i.e., no 
weather display). The pilots’ experience and proficiency were examined in terms of the 
customary indices, such as total flight hours, actual instrument hours, hours “under the 
hood,” flight simulation hours, and hours last 90 days. In addition, a general aviation 
knowledge test was given to the pilots (see Appendix L, General Aviation Questionnaire) 
from which a measure of experience and of their declarative knowledge concerning 
weather were obtained. A descriptive summary of the experience and proficiency 
variables is provided in Table 5.1-1. Note that the participant pilots had a wide range of 
experience and proficiency. Thus, the convenience sample of pilots recruited for the 
experiment was representative of the range of experience and proficiency found in the 
population of GA pilots. 

Table 5.1-1. Subject Pilot Experience and Proficiency Measures 


Experience & 
Proficiency Measures 

Mean 

Standard 

Error 

Max 

Min 

Mean Total Flight 
Hours 

2469.42 

428.77 

15000.00 

220.00 

Mean Actual Instrument 
Hours 

314.44 

105.57 

4000.00 

1.40 

Mean Hours “Under 
The Hood" 

115.25 

24.52 

1000.00 

.00 

Mean Flight Simulation 
Hours 

52.07 

13.00 

300.00 

.00 

Mean Hours Last 90 
Days 

12.91 

4.06 

150.00 

.00 

Weather Knowledge 
Scores 

.66 

2.57E-02 

1.00 

.23 


Although the pilots were randomly assigned to the display type conditions, it is possible 
that simply by chance the pilot characteristics could significantly differ across display 
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conditions. An analysis of variance (ANOVA) indicates there are no significant mean 
differences between the pilots’ characteristics across the three subject pilot groups 
(control, looping NEXRAD, and NCWF) in total hours, actual instrument hours, hours 
“under the hood,” flight simulation hours, hours last 90 days or weather knowledge. The 
groups are similar (i.e., statistically matched) in terms of their experience and proficiency 
as measured by the indices used in this experiment — total flight hours, instrument flight 
experience, currency, and weather knowledge. Thus, the level of experience and 
proficiency of the pilots did not confound the ability to assess the effects of display type 
on pilot performance (e.g., decision quality) in this experiment. 

Although pilot experience did not differ significantly across experimental conditions, 
experience variables might be associated with pilot performance. For instance, it makes 
sense that more experienced pilots may make higher-quality decisions than less 
experienced pilots. The pilot experience and proficiency measures, however, were not 
significantly associated in the appropriate direction with any of the primary dependent 
variables of the experiment (i.e., decision quality metrics) 2 . Feldt (1958) suggested that 
statistically controlling for variables that are associated with dependent measures less 
than r = .20 can actually lead to decreases in statistical power. Thus, it was deemed 
unnecessary to statistically control for the effects of pilot experience and proficiency as 
such analyses would not lead to increases in statistical power to detect the effect of 
display type on the primary dependent measures. 

5.2 Effects of Display Type on Decision Quality 

The results of the experiment are organized around the two key decision points 
established in the experiment procedure — the “Richmond decision” and the “Wallops 
Island decision.” 

Both the Richmond and Wallops Island decisions were scored on a 1-4 ordinal scale, 
with a (1) being a strong “poor” decision, and a (4) being a strong “good” decision. A 
score of (2) was considered a “poor” decision with good elements, while a score of (3) 
was considered a “good” decision with poor elements. The intent was to produce 
definitive guidelines that can be applied to each scenario using a consistent method. 

A good decision — a score of (3) or (4) — was deemed to be one in which the pilot decided 
to divert to Wallops Island prior to the Final Approach Fix (outer marker) of the approach 
into the Richmond airport, thus avoiding the hazardous weather by at least five nautical 
miles. A poor decision — a score of (1) or (2) — was deemed to be one in which the pilot 
continued with an approach past the Final Approach Fix into the Richmond airport for 
whatever reason, placing the aircraft within five nautical miles of hazardous weather 


2 The conclusion that the experience and proficiency measures were not associated with the decision quality 
metrics was based on Pearson correlation analyses and Spearman correlation analyses. Also, the quality of 
pilots’ decisions was used to create post-hoc groups. ANOVA was conducted using the decision-quality 
groupings as the explanatory variable and the experience measures as response variables. The ANOVA 
supported the correlation analyses that there was no statistically significant association of decision quality 
with any of the experience and proficiency measures. That is, pilots who made good decisions were not 
significantly more experienced or proficient than pilots who made poor decisions. 
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conditions. Hazardous weather was established to be a red NEXRAD mosaic image cell, 
a known area of hazardous turbulence, or a known area of hazardous windshear. A 
minimum separation of 5 miles from the most hazardous part of convective weather 
depicted in a NEXRAD image (red cells) as seen on the air traffic controller’s weather 
radar display was selected as the criteria for this segment of the scenario because: 

a. The hazard is a rapidly moving and fairly localized thunderstorm with a well-defined 
leading edge. 

b. The weather conditions five miles and greater to the east of the thunderstorm was 
known to be reasonably safe with no significant turbulence. 

c. The wedding scenario created a motivation to proceed to within a reasonable but 
safe distance. 

The “Richmond decision” required the participant pilot to decide whether or not to 
attempt to land at the Richmond airport in the face of a rapidly moving thunderstorm 
passing within a mile or two to the north of the airport. There were a total of 1 1 different 
NEXRAD mosaic images displayed to the subject pilots who had the NEXRAD Image 
Display, updating in 7-minute intervals. Figures 5.2-la and 5.2-lb depict the NEXRAD 
mosaic image available on the weather display during the period 1921Z to 1925Z as the 
pilots approached the Final Approach Fix to runway 34 to the southeast of the Richmond 
airport. The weather information displayed in the same time frame to the subject pilots 
who had the NCWF Display is depicted in Figures 5.2- lc and 5. 2- Id. 
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Figure 5.2-1 a. NEXRAD Display (with looping) seen by subject pilots 
inbound to Richmond Airport IAF (25nm Scale) 



Figure 5.2-1 b. NEXRAD Display (with looping) seen by subject pilots 
inbound to Richmond Airport IAF (lOnm Scale) 
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Figure 5.2-1c NCWF Display seen by subject pilots inbound to 
Richmond Airport IAF (25nm Scale) 



Figure 5.2-1 d NCWF Display seen by subject pilots inbound to 
Richmond Airport IAF (lOnm Scale) 
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Because of the delay in transmission of the image to the aircraft, the data was at least 
seven minutes old and could have been as many as 14 minutes old. Actual conditions at 
the Richmond airport in the time frame of this decision can be seen in Figures 5.2-2 and 
5.2-3, which depict the NEXRAD mosaic images with the time stamps of 192 1Z and 
1928Z respectively. 



Figure 5.2-2. Actual Conditions at Richmond Airport (as seen in 
NEXRAD Mosaic Image with 1921Z time stamp) 
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Figure 5.2-3. Actual Conditions at Richmond Airport (as seen in 
NEXRAD Mosaic Image with 1928Z time stamp) 

The thunderstorm seen to the northwest of Richmond is the storm that was designed to 
elicit a weather decision from the pilots. This particular storm moved from west to east 
across the successive NEXRAD images at approximately 40 nautical miles per hour in 
the early images. The rate of movement of the storm diminished to less than 10 nautical 
miles per hour in the later images. 

The four-point grading criteria of the Richmond decision are defined as follows: 

1 = The pilot continued the approach into poor weather and was waved off the 

approach (from the tower controller) at the Final Approach Fix (outer marker). 

2 = The pilot abandoned the approach less than five (5) miles outside of the outer 

marker, but flew within five (5) miles of a red NEXRAD image cell, while in 
the Richmond area. 

3 = The pilot abandoned the approach by their own decision less than five (5) miles 

outside of the outer marker, and flew more than five (5) miles from a red 
NEXRAD image cell, while in the Richmond area. 
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4 = The pilot abandoned the approach more than five (5) miles outside of the outer 
marker, and flew more than five (5) miles from a red NEXRAD image cell. 


The “Wallops Island decision” required the participant pilot to decide whether to proceed 
as first cleared to Wallops Island or detour around the hazardous weather. To proceed as 
cleared, the pilot would have flown between two thunderstorms located between 
Richmond and Wallops Island. Figures 5.2-4a and 5.2-4b provide the images that were 
available on the NEXRAD Display as the pilots departed the vicinity of the HARCUM 
VOR enroute to Wallops Island. 



Figure 5.2-4a. NEXRAD Display (with looping) seen by subject pilots 
along route to Wallops at decision time (25nm Scale) 
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Figure 5.2-4b. NEXRAD Display (with looping) seen by subject pilots 
along route to Wallops at decision time (lOnm Scale) 



Figure 5.2-4c. NCWF Display seen by subject pilots along route to Wallops 
at decision time (25nm Scale) 
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Figure 5.2-4d. NCWF Display seen by subject pilots along route to 
Wallops at decision time (lOnm Scale) 

In these images, there is a line of convective activity over the Chesapeake Bay, between 
Richmond and the Wallops Island airport. Within this line of convective activity are two 
thunderstorm cells that did not move significantly in position, but that changed shape and 
size slightly between NEXRAD images. 

A four-point scale was again used to grade the pilot decisions relating to the 
thunderstorms over the Chesapeake Bay. A good decision — score of (3) or (4) — was 
deemed to be one in which the pilot circumvented the hazardous area entirely by 
changing course to the south, so as to avoid it by at least ten nautical miles as determined 
from the air traffic controller’s weather radar display which displayed the weather returns 
in near real time. The pilot would then proceed up the coast of Virginia to the Wallops 
Island airport. A poor decision — score of (1) or (2) — was deemed to be one in which the 
pilot decided to find his way around (or through) the thunderstorms in an attempt to 
proceed by the most direct route to the Wallops Island airport, and for whatever reason, 
flew within ten nautical miles of hazardous weather. 

The four-point grading criteria of the Wallops decisions are defined as follows: 

1 = The pilot flew within 10 miles of a red cell while circumventing the storms over 
the bay using the pilot’s own route planning. 
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2 = The pilot flew within ten miles of a red cell, but only because of a delayed turn 

or distraction, but the intent was to circumvent by at least 10 miles. 

3 = The pilot flew within 10 miles of a red cell, but was following vectors from ATC 

and for whatever reason, ATC vectored the pilot to within that 10 miles. 

4 = The pilot avoided a red cell by 10 miles or more. 

An attempt to take a route to the north around the convective activity was also deemed to 
be a poor decision as the convective area continued into extensive restricted airspace that 
was in use and not available to the pilot. 

In the following subsections, each figure is followed by the data and relevant analysis 
that led to the diagrams. 

5.2.1 Decision Quality: Observer Ratings 

Average decision quality for each type of display (i.e., NCWF, Looping NEXRAD, and 
Control) is plotted by decision type (i.e., Richmond or Wallops) in Figure 5.2. 1-1. A 
repeated measures ANOVA was conducted on the 4-point decision quality metrics. The 
ANOVA indicates that there is a significant main effect of decision type, F(l,43) = 15.75, 
p < .001. Pilots made significantly poorer quality decisions at Richmond than at Wallops. 
There was no effect of type of display on decision quality, F(2,43)=.06, p < .95. That is, 
pilots with weather displays did not make significantly higher-quality decisions than 
pilots without weather displays. Although the interaction between decision type and type 
of display is not statistically significant, F(2,43)=1.22, p < .31, Figure 5.2. 1-1 
reveals some monotone interaction. For example, NCWF led to the highest-quality 
decisions in the Richmond leg (i.e., temporal decision), but the poorest-quality decisions 
in the Wallops leg (i.e., spatial decision) when compared to the other displays. Moreover, 
pilots with looping NEXRAD made the highest-quality decisions at Wallops, but the 
poorest-quality decisions at Richmond when compared to the other displays. 
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Mean Observer Ratings of Decison Quality by 
Decision Type and Display Type 
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Figure 5.2.1 -1. Observer Ratings of Decision Quality by Decision 
Type and Display Type 


Within-Subjects Factors 


General Linear Model 

Between-Subjects Factors 


Measure: MEASURE 1 


N 


Dependent 


CONDITIO Control 

15 

DTYPE 

Variable 


Looping 

16 

1 

RICDQ 


NCWF 

15 

2 

WALDQ 




Descriptive Statistics 


CONDITIO 

Mean 

Std. Deviation 

N 

RICDQ Control 

2.13 

1.41 

15 

Looping 

1.81 

1.33 

16 

NCWF 

2.40 

1.45 

15 

Total 

2.11 

1.39 
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WALDQ Control 

3.27 

1.03 

15 

Looping 

3.37 

1.15 

16 

NCWF 

2.93 

1.33 

15 

Total 

3.20 

1.17 
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5.2.2 Decision Quality: Minimum Distance from Red Cell 

The air traffic controller and experiment observers recorded the minimum distance that 
each pilot flew from a red cell on both the Richmond leg and the Wallops leg. The 
minimum distance from red cell metric provides another measure of decision quality. 
Average minimum distance from red cell for each type of display (i.e., NCWF, Looping 
NEXRAD, and Control) is plotted by decision type (i.e., Richmond or Wallops) in Figure 
5.2.2- 1. A repeated measures ANOVA was conducted on the minimum distance from red 
cell metric. The ANOVA indicates that there is no significant main effect for either 
decision type, F(l,34) = 00.17, p > .68, or type of display, F(2,34) = 00.03, p > .97, on 
the minimum distance from red cell. Although the interaction between decision type and 
type of display is marginally statistically significant, F(2,34)=1.82, p<.10., Figure 5.2.2-1 
clearly reveals some non-monotone interaction. The interaction implies that a particular 
type of display (e.g., NCWF) had differential effects on minimum red cell distance 
depending on the type of decision being made by the pilots. For example, pilots with 
NCWF had longer minimum distances from red cell on the Richmond leg (i.e., temporal 
decision) than on the Wallops leg (i.e., spatial decision). Pilots with Looping NEXRAD 
and pilots without a weather display had longer minimum distances from red cell on the 
Wallops leg than on the Richmond leg. 


Mean Minimum Distance from Red Cell by 
Decision Type and Display Type 
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Figure 5.2.2-1. Minimum Distance from Red Cell by 
Decision Type and Display Type 
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General Linear Model 


Within-Subjects Factors 


Measure: M 

EASURE 1 

DECTYPE 

Dependent 

Variable 

1 

REDRIC 

2 

REDWALL 


Between-Subjects Factors 



N 

DISPLAYS Control 

14 

Looping 

12 

NCWF 

11 


Descriptive Statistics 


DISPLAYS 

Mean 

Std. Deviation 

N 

REDRIC Control 

8.500 

7.367 

14 

Looping 

7.000 

6.075 

12 

NCWF 

11.000 

9.455 

11 

Total 

8.757 

7.635 

37 

REDWALL Control 

9.714 

4.065 

14 

Looping 

10.917 

3.630 

12 

NCWF 

7.727 

3.849 

11 

Total 

9.514 

3.969 
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5.3 Findings from the Immediate Reactions Questionnaire 

Upon completion of the simulator session, each participant pilot was given a 
questionnaire (Appendix J, Immediate Reactions Questionnaire) to obtain his or her 
immediate reactions. The pilot was given the questionnaire while still seated in the 
simulator, thereby reducing distraction issues and obtaining valuable subjective 
information while the pilot was still in the “flight mode.” 

Except for questions 10 and 11, there were five possible answers on a Likert scale that 
ranged from Disagree (score of 1) to No Opinion to Agree (score of 5). This 
questionnaire was reviewed with the pilot during the post-flight briefing to verify that the 
pilot understood the questions and to clarify any ambiguous answers. 

The questionnaire responses are presented here in their entirety, but correlations to 
relevant data are included in succeeding sections. 

Question 1. This question was asked to determine the extent to which the participant 
pilots “bought into” the wedding scenario. All pilots were asked this question. Also, there 
were no significant differences between pilots’ responses to this question across the 
display conditions. 

I took the wedding scenario as a serious personal commitment, in the 
sense that I factored the time pressure into my decision-making. 
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(mean score of 4.04, standard deviation of 1.17) 

A score of four (4) was indicative of an “agree somewhat. ” A mean score of 4.04 and 
standard deviation of 1.17 with this question signifies that the majority of pilots 
somewhat “bought into ” the wedding scenario, thus the scenario was sufficiently 
realistic to engage the pilots in the situation. 

Question 2 . This question explored the extent to which the pilots perceived the weather 
information display as an aid to make better decisions. Only participant pilots who flew 
with a weather display received this question. There was not a significant difference in 
response to the question between pilots who flew with the NCWF display and pilots who 
flew with NEXRAD Image Looping display. 

An advantage of the onboard Weather Display was showing the 
graphical depiction of the weather to aid in my decision-making. 

(mean score of 4.61, standard deviation of .56) 

A score of five (5) was indicative of an “agree. ” A mean score of 4.61 and standard 
deviation of .56 with this question signifies that the majority of pilots perceived the 
NCWF display and NEXRAD image Looping display as an advantage to their decision 
making. 

Question 3 . This question explored the extent to which the pilots perceived the weather 
information display as increasing their situational awareness. Only participant pilots who 
flew with a weather display received this question. There was not a significant difference 
in the average response to the question between pilots who flew with the NCWF display 
and pilots who flew with the NEXRAD Image Looping display. 

The weather display increased my situational awareness during the 
decision-making process. 

(mean score of 4.44, standard deviation of .91) 

A mean score of 4.44 and standard deviation of .91 on this question indicates that the 
pilots generally perceived the weather display as increasing their situational awareness. 

Question 4 . This question explored the extent to which the pilots perceived themselves as 
attempting to use all the sources of weather information available to them. This question 
was asked of all pilots. There were no significant differences in the average response to 
the question across the three display conditions. 

I tried to systematically sample all sources of weather information 
open to me. 

(mean score of 3.37, standard deviation of 1.35) 

The mean score of 3.37 indicates that the perceived strategy of the pilots was not to 
systematically sample all possible sources of weather information. However, the 
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relatively large standard deviation of 1.35 suggests a wide variation in the pilot’s 
perceived strategies of sampling all possible sources of weather information. 

Question 5 . This question explored the extent to which pilots believed it was necessary 
to crosscheck their inferences from the cockpit weather information display with 
conventional weather data sources. Only participant pilots who flew with a weather 
display received this question. There was not a significant difference in the average 
response to the question between pilots who flew with the NCWF display and pilots who 
flew with the NEXRAD Image Looping display. 

I used the weather display, but felt the need to crosscheck or verify 
my conclusions from conventional weather data sources (ATC, etc.). 

(mean score of 4.03, standard deviation of 1.15) 

The mean score of 4. 03 indicates that pilots generally perceived a need to crosscheck 
their inferences when those inferences were based on information from the weather 
display. 

Question 6 . This question explored the extent to which the pilots perceived themselves 
as understanding and appropriately operating the autopilot. The question was given to all 
pilots. Also, there were no significant differences in the average response to the question 
across the three display conditions. 

I felt comfortable with the autopilot, in terms of understanding its use 
and operation. 

(mean score of 4.17, standard deviation of 1.15) 

The mean score of 4.17 indicates that pilots generally perceived themselves as being 
somewhat comfortable with the autopilot. 

Question 7 . This question explored the pilots’ perceived ability to successfully complete 
the flight without the aid of the autopilot. The question was given to all pilots. There were 
significant differences in the average response to the question across the three display 
conditions, F(2,44)=4.49, p = .02. 

Without the autopilot, my completion of the flight would have been 
compromised. 

Table 5.3-1. Pilots’ Perception of Their Ability To 


Complete Mission Wjthout Autopilot 


Condition 

Mean Score 

Standard Deviation 

Control 

4.13 

1.31 

NCWF Display 

3.47 

1.55 

NEXRAD Looping Display 

4.75 

.45 

Total 

4.13 

1.28 
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The mean scores indicate that pilots in the Control and NEXRAD Image Looping 
conditions considered the autopilot necessary to the safe completion of the flight. The 
autopilot may have decreased workload and allowed the pilots more time to sample 
weather information sources. Pilots who flew with the NCWF display, however, believed 
the autopilot was less necessary for the safe completion of the flight than pilots in the 
other conditions. Pilots with the NCWF display may have been less compelled to search 
for conventional sources of weather information because the NCWF product is an 
integrative tool, which includes multiple weather products. For this reason pilots with the 
NCWF display may have perceived reductions in workload provided by the autopilot as 
less essential for the safe completion of the flight than pilots in the other display 
conditions. 

Question 8 . This question explored the extent to which pilots’ perceived themselves as 
attending to the latency and accuracy of the weather information. Only participant pilots 
who flew with a weather display received this question. There was not a significant 
difference in the mean response to the question between pilots who flew with the NCWF 
display and pilots who flew with the NEXRAD Image Looping display. 

The degree of validity and timeliness of the weather data appearing 
on the display was a factor I felt that I held in mind as I flew. 

(mean score of 4.39, standard deviation of 1.05) 

The mean score of 4.39 indicated that the pilots believed they were aware of the age of 
the information depicted on the cockpit weather display. 

Question 9 . This question explored the pilots’ perceptions of whether they attended to 
the age display, which provides a graphical representation of the age of the weather 
information, in their instrument scans. Only participant pilots who flew with a weather 
display received this question. There was not a significant difference in the mean 
response to the question between pilots who flew with the NCWF display and pilots who 
flew with the NEXRAD Image Looping display. 

I have been monitoring the weather age display regularly in my 
instrument scan. 

(mean score of 4.28, standard deviation of 1.17) 

The mean score of 4.28 indicated that the pilots believed they were including the age 
display in their instrument scan. Thus, the ratings of Questions 9 and 10 suggest that the 
pilots were both aware and regularly sampled (i.e., utilized) the age display. 

Question 10 . The pilots were asked what they perceived the weather conditions to be near 
the Richmond airport. The question was given to all pilots. Also, there were no 
significant differences in the number of response selections to the question across the 
three display conditions. 
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At the time of my arrival to the Richmond airport, I knew that there 
was a storm-- 

a. About 10 nm northwest of the airport. (7 selections) 

b. About 5 nm northwest of the airport. (6 selections) 

c. Near the airport. (19 selections) 

d. Right at the airport. (16 selections) 

Most of the pilots correctly perceived that the storms were very close to the airport. This 
perception, however, did not necessarily contribute to good-quality decisions. 

Question 1 1 . The pilots were asked what they perceived the weather conditions to be 
enroute to Wallops Island. The question was given to all pilots. Also, there were no 
significant differences in the number of response selections to the question across the 
three display conditions. 

At the time I was en-route to Wallops Island, I saw across my path of 
direct flight, what I took to be — 

a. A penetrable storm. (1 selection) 

b. A navigable opening between convective cells. (14 selection) 

c. A non-navigable opening between cells. (5 selection) 

d. A wall of convective activity requiring diversion. (18 selection) 

Although the most pilots (18) correctly perceived the storms as requiring diversion, there 
were several pilots (15) who perceived the storm as navigable or penetrable. It is 
important to note that the participant pilots who perceived the storm as requiring 
diversion did not necessarily make good-quality decisions. 

Question 12 . The question assessed the extent to which the pilot perceived that the 
ownship icon provided an accurate representation of aircraft position. Only participant 
pilots who flew with a weather display received this question. There was a marginally 
statistically significant difference in the mean response to the question between pilots 
who flew with the NCWF display and pilots who flew with the NEXRAD Image Looping 
display, F(l,30)=2.90, p =.06. 

On the weather display, I found the positional accuracy of the aircraft 
icon to be adequate. 


Table 5.3-2. Pilots’ Perception that the Ownship Icon Provided an 
Accurate Representation of Aircraft Position 


Condition 

Mean Score 

Standard Deviation 

NEXRAD Looping Display 

4.27 

.96 

NCWF Display 

3.25 

1.77 

Total 

3.74 

1.50 
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The mean scores indicate that pilots with the NEXRAD Looping display considered the 
positional accuracy of the ownship icon to be adequate. The mean score of NCWF pilots 
indicate they had no opinion as to the positional accuracy of the ownship icon. The 
relatively large standard deviation of the NCWF condition, however, suggest that the 
pilots who flew with the NCWF display had varying opinions as to the positional 
accuracy of the ownship icon. 

Question 13 . This question assessed the extent to which the pilots perceived the weather 
display as helping them to identify their aircraft position relative to any storms. Only 
participant pilots who flew with a weather display received this question. There was not a 
significant difference in the mean response to the question between pilots who flew with 
the NCWF display and pilots who flew with the NEXRAD Image Looping display. 

In using the weather display, I felt that I precisely knew the aircraft 
position relative to any storms. 

(mean score of 3.47, standard deviation of 1.39) 

The mean score of 3.47 indicated that the pilots generally did not believe they precisely 
knew their relative position to storms when relying on the weather display. However, the 
relatively large standard deviation of 1.39 suggests a wide variation in the pilots ’ 
perceptions of how precisely relative position to storms was known. 

Question 14 . The purpose of this question was to determine if the pilots believed they 
had enough valid information to make confident decisions concerning hazardous weather. 
The question was given to all pilots. There were significant differences in the average 
response to the question across the three display conditions F(2,44)=3.91, p < .03. 

I felt that I had adequate sources of weather information to make 
confident decisions. 


Table 5.3-3. Pilots’ Perception that They Had Adequate Weather 
Information To Make Confident Decisions 


Condition 

Mean Score 

Standard Deviation 

Control 

2.93 

1.53 

NCWF Display 

4.06 

1.31 

NEXRAD Looping Display 

4.13 

1.12 

Total 

3.72 

1.41 


The mean scores indicate that pilots with the NCWF display and the NEXRAD Image 
Looping display felt they had adequate sources of weather information to make confident 
decisions. Pilots without a weather display, however, felt they had relatively inadequate 
sources of weather information to make confident decisions. 
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Question 15 . This question assessed the extent to which the pilots perceived that the 
presence of the weather display increased their head-down time and scanning workload. 
In other words, this question partially captures the extent to which the pilots perceived 
they cognitively tunneled on the weather display. Only participant pilots who flew with a 
weather display received this question. There was not a significant difference in the mean 
response to the question between pilots who flew with the NCWF display and pilots who 
flew with the NEXRAD Image Looping display. 

The weather display increased head-down time and scanning 
workload. 

(mean score of 3.56, standard deviation of 1.34) 

The mean score of 3.56 indicates that the pilots were generally unaware of any influence 
the weather display may have had on head-down time or scanning workload. However, 
the relatively large standard deviation of 1.48 suggests a wide variation in the perception 
of the effects of the displays on head-down time and scanning workload. 

Question 16 . This question explored the pilot’s perception of the latency of the weather 
information, and the extent to which the weather display was showing depicted weather 
in “real time.” Only participant pilots who flew with a weather display received this 
question. There was not a significant difference in the mean response to the question 
between pilots who flew with the NCWF display and pilots who flew with the NEXRAD 
Image Looping display. 

I was aware of the latency of the weather data appearing on screen as 
I flew and assessed alternative flight paths. 

(mean score of 4.23, standard deviation of 1.12) 

The mean score of 4.23 indicated that pilots were generally aware of the latency of the 
weather information. 

Question 17 . This question tapped the pilot’s perception of the usefulness of the range- 
rings in determining their proximity to weather cells. Only participant pilots who flew 
with a weather display received this question. There was not a significant difference in 
the mean response to the question between pilots who flew with the NCWF display and 
pilots who flew with NEXRAD Image Looping display. 

I found the range rings very useful in enhancing my weather cell 
awareness. 

(mean score of 3.63, standard deviation of 1.48) 

The mean score of 3. 63 indicates that the pilots generally had no opinion one way or 
another as to the usefulness of the range-rings. However, the relatively large standard 
deviation of 1.48 suggests a wide variation in the perception of the usefulness of the 
range rings among the pilots. 
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Question 18 . This question tapped the pilot’s perception of the flight-path waypoints on 
decision making. Only participant pilots who flew with a weather display received this 
question. There was not a significant difference in the mean response to the question 
between pilots who flew with the NCWF display and pilots who flew with the NEXRAD 
Image Looping display. 

The presentation of flight path waypoints facilitated my decision- 
making. 

(mean score of 3.91, standard deviation of 1.38) 

The mean score of 3.91 indicates that the pilots generally perceived the depiction of 
waypoints on the weather display as facilitating decision-making. However, the relatively 
large standard deviation of 1.38 suggests a wide variation in the perception of the 
usefulness of the waypoints on decision-making among the pilots. 


5.4 Findings from the Structured Interview 

The subject pilots’ answers to several key questions asked in the structured interview that 
are related to their decision making processes will be considered in this section. 

Question 3 . This Question was asked to determine if the pilots believed they had enough 
information to make an accurate and confident decision during the Richmond leg of the 
flight. 

Do you feel that you had enough information to make a sound and 
confident judgment of the situation? 

Pilot responses to the question are plotted by display type in Figure 5.4-1. Logistic 
ANOVA was conducted on the pilot responses. A logistic ANOVA Likelihood Ratio test 
indicated that the type of display influenced the likelihood that pilots perceived 
themselves as having enough information on the Richmond Leg of the flight ( )f(2,45)= 
16.91, p <.001). The results indicate control pilots were significantly less likely to 
respond that they had enough information than pilots with the NEXRAD Image Looping 
display or the NCWF display. Also, few pilots who flew with a weather display responded 
that they did not have enough information. 
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Structured Interview Question #3 



■ Not Enough Information 
□ Enough Information 


Display Type 


Figure 5.4-1. Pilots’ Perception that They Had Sufficient Information 
to Make Confident Decisions on Richmond Leg 


Question 7 . This question explored the extent to which the pilots believed they had 
enough time to get all the information they needed to make a decision in the Richmond 
leg of the flight. This question captured workload by tapping into the extent pilots 
perceived they were under time pressure. 

Do you feel that you had enough time to gather all the information 

that you wanted? 

Pilot responses to the question are plotted by display type in Figure 5.4-2. Logistic 
ANOVA was conducted on the pilot responses. A logistic ANOVA Likelihood Ratio test 
indicated that the type of display did not influence the pilots perception of whether they 
had enough time on the Richmond leg to gather all the information they wanted 
(X(2,45)= 1.30, p < .52). The results indicate that the majority of pilots across all 
conditions believed they had enough time to gather information on the Richmond leg of 
the flight. 
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Structured Interview Question #7 
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Figure 5.4-2. Pilots’ Perception that They Had Sufficient Time to 
Collect Information They Wanted On Richmond Leg 


□ Not Enough Time 

□ Enough Time 


Question 1 1 . This Question was asked to determine if the pilots believed they had enough 
information to make an accurate and confident decision during the Wallops leg of the 
flight. 

Do you feel that you had enough information to make a sound and 
confident judgment of the situation? 

Pilot responses to the question are plotted by display type in Figure 5.4-3. Logistic 
ANOVA was conducted on the pilot responses. A logistic ANOVA Likelihood Ratio test 
indicated that type of display did not influence the pilot responses to this question 
(X(2,41)= 1.24, p <.54). The results indicate that the majority of pilots in all conditions 
believed they had enough information on the Wallops leg of the flight. 
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Structured Interview Question #11 
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Figure 5.4-3. Pilots’ Perception that They Had Sufficient Information 

to Make Confident Judgment of Situation on Wallops Leg 


n Not Enough Information 
□ Enough Information 


Question 15 . This question explored the extent to which the pilots believed they had 
enough time to get all the information they needed to make a decision in the Wallops leg 
of the flight. This question captures workload by tapping into the extent pilots perceived 
time pressure during the Wallops leg of the flight. 

Do you feel that you had enough time to gather all the information 

that you wanted? 

Pilot responses to the question are plotted by display type in Figure 5.4-4. Logistic 
ANOVA was conducted on the responses. A logistic ANOVA Likelihood Ratio test 
indicated that type of display did not influence the pilot responses to the question 
(Z(2,40)= 2.30, p < .32). The results indicate that the majority of pilots across all 
conditions believed they had enough time on the Wallops leg of the flight. Also, note that 
only one control pilot reported that he did not have sufficient time. 
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Structured Interview Question #15 



H Not Enough Time 
□ Enough Time 


Display Type 


Figure 5.4-4. Pilots’ Perception that They Had Sufficient Time to 
Collect Information They Wanted On Wallops Leg 


Question #22 . This question explored the pilots’ perceptions of the usefulness of the 
Weather Displays in comparison to conventional sources of weather information. 

How would you compare the weather display information to other 
sources of weather information (ATIS, Flight Watch, etc.)? 

Pilot responses to the question are plotted by display type in Figure 5.4-5. Logistic 
ANOVA was conducted on the responses. A logistic ANOVA Likelihood Ratio test 
indicated that type of display did not significantly influence the pilots ’ responses to the 
question (yf( 2,27)= 1.50, p < .48). The results indicate that the majority of pilots 
believed the weather display was more useful than traditional sources of weather 
information. 
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Structured Interview Question #22 


16 , 



Better Same Worse 

Wx Display Compared to Other Wx Sources 
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Figure 5.4-5. Pilots’ Perception As To Usefulness of Weather Display 
Information To Other Sources of Weather Information 
(ATIS, Flight Watch, etc.) 

Question #23 . This question explored the extent to which the pilots believed they used 
the weather display for navigation. 

Did you use the weather display to help you navigate? 

Pilot responses to the question are plotted by display type in Figure 5.4-6. Logistic 
ANOVA was conducted on the responses. A logistic ANOVA Likelihood Ratio tests 
indicate that type of display significantly influenced the pilots ’ responses to the question 
(1,29)= 2.85, p< .10). The results indicate that a relatively large number of pilots 
with the NEXRAD Image Looping display responded that they used the weather display 
for navigation. Moreover, the majority of pilots who flew with a weather display 
responded that they used the weather display for navigation. 
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Figure 5.4-6. Pilots’ Perception as to Whether Or Not They Used 
Weather Display to Navigate 
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Question #24 . This question explored pilot perceptions of how the weather display 
influenced workload. 

Do you feel that the weather display increased or decreased your 
workload? 

Pilot responses to the question are plotted by display type in Figure 5.4-7. Logistic 
ANOVA was conducted on the responses. A logistic ANOVA Likelihood Ratio test 
indicated that type of display did not significantly influence pilot responses to the 
question (yf(2,26)= 2.53, p < .29). Figure 5.4-7 indicates that the majority of pilots who 
flew with a weather display did respond that the weather display increased their 
workload. 
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Figure 5.4-7. Pilot’s Perception of the Effect of Use of Weather 
Display On Workload 

Question #25 . This question explored the pilots’ perceived trust in the accuracy of the 
information presented on the weather display. 


■ Looping 
□ NCWF 


Did you trust the weather display to give you correct information? 

Pilot responses to the question are plotted by display type in Figure 5.4-8. Logistic 
ANOVA was conducted on the responses. A Logistic ANOVA Likelihood Ratio test 
indicated that type of display significantly influenced pilot responses to the question 
tf(l,30)=2.91,p< .09). The results indicate that all pilots with NEXRAD Looping and 
most of the pilots with NCWF responded that they had confidence in the reliability of the 
information presented by the display. 
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Structured Interview Question #25 
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Figure 5.4-8. Pilots’ Perception As Whether They 
Could Trust Weather Display 

Question #26 . This question explored the extent to which the pilots perceived confusion 
concerning ownship symbol. 


Was there any confusion with the ownship symbology? 

Pilot responses to the question are plotted by display type in Figure 5.4-9. Logistic 
ANOVA was conducted on the responses. A logistic ANOVA Likelihood Ratio test 
indicated that type of display did not significantly influence pilot responses to the 
question (yf(l,29)= .01, p < .92). Figure 5.4-9 indicates that the majority of pilots 
responded that there was no confusion concerning the ownship symbol. 


Structured Interview Question #26 
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Figure 5.4-9. Pilots’ Perception As to Whether They Had Any 
Confusion in Use of Ownship Symbology 
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Question #28 . This question explored whether pilot workload due to flying the aircraft 
was within comfort zone. 


Did you feel ahead of the airplane? 

Pilot responses to the question are plotted by display type in Figure 5.4-10. Logistic 
ANOVA was conducted on the responses. A Logistic ANOVA Likelihood Ratio test 
indicated that the type of display did not significantly influence pilot responses to the 
question (yf( 2,28)= .54, p < .77). Figure 5.4-10 indicates that the majority of pilots 
responded that they were behind the airplane. Thus, the workload necessary to fly the 
airplane took the majority of pilots out of their comfort zones. 
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Figure 5.4-10. Pilots’ Perception of Whether They Were 
Ahead or Behind Airplane 


5.5 Effects of Display Type on Workload 

A subjective evaluation of the workload of the subject pilots was undertaken by the 
observers through their observations made during the simulated flight, and was obtained 
from each of the subject pilots upon completion of the flight. 

In the following subsections, each figure is followed by the data and relevant analysis 
that led to the diagrams. 

5.5.1 Observers’ Assessment of Pilot Workload 

The observers made a workload rating on a 100 mm Likert scale for each participant pilot 
on each leg of the flight. The Likert scale had two labels “Extremely Low” and 
“Extremely High” that anchored the scale (see Appendix I, Observer Forms). The 
observer workload ratings were combined for the following analyses. 
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Average workload ratings for each type of display (i.e., NCWF, Looping NEXRAD, and 
Control) are plotted by decision type (i.e., Richmond or Wallops) in Figure 5.5. 1-1. 
Repeated measures ANOVA was conducted on the observer workload ratings. The 
ANOVA indicates that there is a significant main effect of decision type, F(l,40) = 12.95, 
P < .001. Thus, the pilots were observed experiencing significantly higher workload in 
the Richmond leg than in the Wallops leg. There was no effect of type of display on the 
observer ratings of workload, F(2,40)=.167, p > .84. That is, pilots with weather displays 
were not observed experiencing significantly higher workload than pilots without weather 
displays. Also, the decision type by display type interaction was not statistically 
significant, F(2,40)=1.38, p>.15. Note that the general pattern of results from the 
combined observer workload ratings is consistent with the results from each observer’s 
workload ratings. 


Mean Observer Ratings of Workload by Decision 
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Figure 5.5.1 -1. Mean Observers’ Ratings of Pilot Workload 

General Linear Model 

Within-Subjects Factors 


Measure: MEASURE 1 



Dependent 

DTYPE 

Variable 

1 

WLRICQ 

2 

WLWALQ 
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Between-Subjects Factors 



N 

CONDITIO Control 

13 

Looping 

15 

NCWF 

15 


Descriptive Statistics 


CONDITIO 

Mean 

Std. Deviation 

N 

WLRICQ Control 

66.3346 

17.0319 

13 

Looping 

67.8433 

20.0625 

15 

NCWF 

66.9900 

16.2121 

15 

Total 

67.0895 

17.4655 

43 

WLWALQ Control 

60.4192 

18.8540 

13 

Looping 

63.1767 

13.7165 

15 

NCWF 

57.2267 

17.9440 

15 

Total 

60.2674 

16.6715 

43 


5.5.2 Subject Pilot Self-Assessment of Workload 

The participant pilots were asked to provide a self-assessment of their workload during 
the simulated flight just before the structured interview was conducted. The NASA TLX, 
a subjective taskload index, was used in this assessment. The questionnaire consisted of 6 
questions with responses elicited using a 19-point Likert scale (see Appendix O, Subject 
Pilots’ Self-Assessment of Workload). Pilots were asked to use the NASA TLX-Rating 
Scale Definitions depicted in Table 5. 5.2-1 in completing the questionnaire. Note that the 
TLX questions tap six subjective workload dimensions (i.e., mental demand, physical 
demand, temporal demand, performance, effort, and frustration). 


Table 5.5.2-1. NASA TLX-Rating-Scale Definitions 


Title 

Endpoints 

Descriptions 

Mental Demand 

Low/High 

How much mental and perceptual activity was required 
(e.g., thinking, deciding, calculating, remember, 
looking, searching, etc.)? Was the task easy or 
demanding, simple or complicated, exacting or 
forgiving? 

Physical Demand 

Low/High 

How much physical activity was required (e.g., 
pushing, pulling, turning, controlling, activating, etc.)? 
Was the task easy or demanding, slow or brisk, slack or 
strenuous, restful or laborious? 

Temporal 

Demand 

Low/High 

How much time pressure did you feel due to rate or 
pace at which the tasks or task elements occurred? Was 
the pace slow and leisurely or rapid and frantic? 
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Performance 


Good/Poor How successful do you think you were in 

accomplishing the experimenter (or yourself)? How 
satisfied were you with your performance in 
accomplishing this these goals? 

Effort Low/High How hard did you have to work (mentally and 

physically) to accomplish your level of performance? 


Frustration Low/High How insecure, discouraged, irritated, stressed and 

annoyed versus secure, gratified, content, relaxed and 
complacent did you feel during the task? 


The average ratings on the six TLX subjective workload dimensions for each type of 
display are plotted in Figure 5. 5.2-1. As Figure 5. 5.2-1 illustrates, the pilots with the 
NCWF product rated themselves as experiencing higher workload than control pilots or 
pilots with looping NEXRAD on four of the six TLX dimensions. A one-way ANOVA 
indicated that the average rating on the mental demand dimension significantly differed 
across the display type conditions, F(2,45)=4.01, p < .03. Tukey LSD post-hoc tests 
employing Bonferroni alpha adjustments indicated that pilots with NCWF [t(30) = 2.69, 
p < .05] and Control pilots [t(30) = 2.10, p < .05] rated themselves as experiencing 
significantly higher mental demand than pilots with Looping NEXRAD experience. The 
post-hoc analyses also indicated that the average ratings on mental demand did not 
significantly differ between NCWF pilots and Control pilots. Note that the average 
ratings on the other five TLX dimensions did not significantly differ across the display 
conditions. 
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TLX Subjective Workload Dimensions by Display Type 



□ NCWF 
■ Looping 

□ Control 


TLX Dimensions 


Figure 5.5.2-1 . Pilots’ Self Assessment of Workload 
using NASA TLX Rating Scale 

The six TLX dimensions were aggregated to form a composite measure of subjective 
workload. The TLX composite is plotted by display type in Figure 5. 5. 2-2. A one-way 
ANOVA indicates that the mean TLX composite scores do not significantly differ across 
display type. Although the effect is not significant, the overall pattern of TLX results 
suggests that pilots with the NCWF display generally rated themselves as experiencing 
higher workload than control pilots or pilots with the NEXRAD Image Looping display. 
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TLX Composite Index of Subjective Workload by 
Display Type 
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Figure 5.5.2-2. TLX Composite Index of Pilots’ Self-Assessment 
of Workload By Display Type 

Univariate Analysis of Variance 

Descriptive Statistics 

Between-Subjects Factors Dependent Variable: COMPOSIT 


CONDITIO 

Mean 

Std. Deviation 

N 

Control 

76.2500 

12.0083 

16 

Looping 

73.9375 

14.1585 

16 

NCWF 

82.5000 

11.3608 

16 

Total 

77.5625 

12.8277 

48 



N 

CONDITIO Control 

16 

Looping 

16 

NCWF 

16 


Tests of Between-Subjects Effects 


Dependent Variable: COMPOSIT 


Source 

Type III Sum 
of Sauares 

df 

Mean Sauare 

F 

Sia. 

Eta Sauared 

Corrected Model 

627.875 a 

2 

313.938 

1.988 

.149 

.081 

Intercept 

288765.187 

1 

288765.187 

1828.673 

.000 

.976 

CONDITIO 

627.875 

2 

313.937 

1.988 

.149 

.081 

Error 

7105.938 

45 

157.910 




Total 

296499.000 

48 





Corrected Total 

7733.813 

47 






a - R Squared = .081 (Adjusted R Squared = .040) 
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5.5.3 Workload and Decision Quality 

There were no statistically significant associations between the metrics of decision 
quality and the metrics of workload. 

5.6 Effects of Display Type on Pilot Procedures 

It is important to understand how the introduction of cockpit weather information 
displays may influence pilots’ normal operating procedures. For instance, the 
introduction of additional weather information could influence the extent to which pilots 
sample conventional sources of weather information. Introduction of weather displays 
into the cockpit could also increase workload leading to more pilot induced control 
actions. That is, pilots may have a more difficult time flying the airplane when 
interacting with the weather display. Also, pilots who rely on the weather display for 
tactical navigation could experience navigation problems, which could lead to increased 
interactions with NAV radios. Thus, the following sub-sections address the extent to 
which the introduction of weather displays into the cockpit affect pilot procedures. 

In the following subsections, each figure is followed by the data and relevant analysis 
that led to the diagrams. 

5.6.1 Pilot Utilization of Conventional Weather Information Sources 

The observers and air traffic controller kept a record of the conventional weather 
information sources utilized by each pilot for each leg of the flight. The average number 
of weather sources used for each type of display is plotted by decision type (i.e., 
Richmond or Wallops) in Figure 5.6. 1-1. Repeated measures ANOVA was conducted on 
the number of weather information sources utilized. The ANOVA indicates that there is a 
significant main effect of decision type, F(l,45) = 83.35, p < .001. Pilots were observed 
sampling a significantly higher number of conventional weather sources in the Richmond 
leg than in the Wallops leg. There was no effect of type of display on the number of 
weather sources utilized, F(2,45)=2.13, p > .13. That is, pilots with weather displays did 
not sample fewer conventional weather information sources than pilots without weather 
displays. The decision type by display type interaction is statistically significant, 
F(2,45)=4.44, p<.02. The interaction implies that a particular type of display (e.g., 

NCWF display) had differential effects on the number of conventional weather 
information sources utilized depending on the type of decision being made by the pilots. 
For example, pilots with the NCWF display sampled the largest number of conventional 
weather sources during the Richmond leg (i.e., temporal decision), but sampled the 
fewest conventional weather sources during the Wallops leg (i.e., spatial decision). 
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Mean Number of Conventional Weather 
Information Sources Used by Decision Type and 

Display Type 
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Figure 5.6.1 -1. Mean Number of Conventional Weather Information 
Sources Used by Decision Type and Display Type 


General Linear Model 


Within-Subjects Factors 


Measure: MEASURE 1 



Dependent 

DTYPE 

Variable 

1 

NUMWXRIC 

2 

NUMWXWAL 


Between-Subjects Factors 



N 

CONDITIO Control 

16 

Looping 

16 

NCWF 

16 


Descriptive Statistics 


CONDITIO 

Mean 

Std. Deviation 

N 

NUMWXRIC Control 

4.0625 

1.9138 

16 

Looping 

2.9375 

1.6520 

16 

NCWF 

4.4375 

1.7500 

16 

Total 

3.8125 

1.8526 

48 

NUMWXWAL Control 

2.0625 

1.2366 

16 

Looping 

1.5000 

.8944 

16 

NCWF 

1.3125 

1.0145 

16 

Total 

1 .6250 

1 .0842 

48 
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The number of pilots who utilized each of the weather information sources available at 
Richmond (as noted by the observers) is plotted by display type in Figure 5.6. 1-2. 
Logistic ANOVAs were conducted on the number of pilots who utilized each weather 
information source during the Richmond leg. Logistic ANOVA Likelihood Ratio tests 
indicate that the type of weather display did not influence the likelihood that pilots would 
utilize PHF ATIS, RIC ATIS, WAL ASOS, ATC, Tower, Approach Control, or 
Departure Control during the Richmond leg of flight. Logistic ANOVA Likelihood Ratio 
tests did indicate that type of display influenced the probability that pilots would utilize 
FSS/FW (x 2 (2,45)= 5.29, p <.07), PIREP (x 2 (2,45)= 5.14, p <.08), METAR (x 2 (2,45)= 
23.40, p <.00 1), and Wx Display (x 2 (2,45)= 54.92, p <.00 1). Pilots with NEXRAD 
Looping were significantly less likely to utilize FSS/FW and PIREPs than pilots with 
NCWF or control pilots. Also, control pilots did not utilize METARs or Wx Displays 
because they did not have access to this information. 

Observer Noted RIC Wx Sources Utilized 
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Figure 5.6.1 -2. Weather Information Sources Used On Richmond 
Leg as Recorded by Observers 

The number of pilots who used each of the weather information sources available at 
Wallops (as noted by the observers) is plotted by display type in Figure 5.6. 1-3. Logistic 
ANOVAs were conducted on the number of pilots who utilized each weather information 
source during the Wallops leg. Logistic ANOVA Likelihood Ratio tests indicate that the 
type of weather display did not influence the likelihood that pilots would utilize PHF 
ATIS, RIC ATIS, Tower, Approach Control, Departure Control, or PIREPs during the 
Wallops leg of flight. Logistic ANOVA Likelihood Ratio tests did indicate that type of 
display influenced the probability that pilots would utilize WAL ASOS (x 2 (2,43)= 6.17, p 
<.05), FSS/FW (x 2 (2,43)= 14.30, p <.001), ATC (x 2 (2,43)= 4.69, p <.10), METAR 
(X 2 (2,43)= 24.85, p <.001), and Wx Display (x 2 (2,43)= 25.99, p <.001). The results 
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indicate that pilots with NCWF were significantly less likely to utilize WAL ASOS or 
ATC than pilots with NEXRAD Looping or control pilots during the Wallops leg of the 
flight. Pilots with NCWF, however, were significantly more likely to utilize FSS/FW 
than pilots with NEXRAD Looping or control pilots during the Wallops leg of flight. 
Also, control pilots did not utilize METARs or Wx Displays to the same extent as the 
other pilots because they did not have access to this information. 
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Figure 5.6.1 -3. Weather Information Sources Used On 
Wallops Leg as Recorded by Observers 


Question # 4 in the structured interview asked the participant pilots to identify their 
primary and secondary sources of weather information in the Richmond leg of flight. The 
number of pilots who identified each of the weather information sources available at 
Richmond as primary is plotted by display type in Figure 5.6. 1-4. Logistic ANOVAs 
were conducted on the number of pilots who identified each weather information source 
as primary in the Richmond leg of flight. Logistic ANOVA Likelihood Ratio tests 
indicated that the type of weather display did not influence the likelihood that pilots 
would identify ATIS, ATC, FSS/FW, PIREP, METAR, or a combination (i.e., Combo) of 
weather sources as primary in the Richmond leg of flight. Logistic ANOVA Likelihood 
Ratio tests did indicate that type of display influenced the probability that pilots would 
identify Pre-Flight Briefing (x(2,44)= 1 1.64, p <.01) and Wx Display (x 2 (2,44)= 16.23, p 
<.00 1) as primary in the Richmond leg of flight. The results indicate that control pilots 
were significantly more likely to identify the Pre-Flight Briefing as primary during the 
Richmond leg of flight than pilots with the NEXRAD Image Looping display or NCWF 
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display. Also, control pilots did not identify Wx Displays as primary because they did not 
have access to this information. 
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Figure 5.6.1 -4. Pilots’ Self-Assessment of the Primary Source of 
Weather Information Used on Richmond Leg 

The number of pilots who identified each of the weather information sources available at 
Richmond as secondary is plotted by display type in Figure 5.6. 1-5. Logistic ANOVAs 
were conducted on the number of pilots who identified each weather information source 
as secondary in the Richmond leg of flight. Logistic ANOVA Likelihood Ratio tests 
indicated that type of display influenced the probability that pilots would identify Wx 
Display (x 2 (2,44)= 16.23, p < .001) as secondary in the Richmond leg of flight. Control 
pilots did not identify Wx Displays as a secondary source of weather information because 
they did not have access to this information. 
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Secondary Wx Sources RIC 
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Figure 5.6.1 -5. Pilots’ Self-Assessment of the Secondary Source of 
Weather Information Used On Richmond Leg 


Question # 12 in the structured interview asked the participant pilots to identify their 
primary and secondary sources of weather information in the Wallops leg of flight. The 
number of pilots who identified each of the weather information sources available at 
Wallops as primary is plotted by display type in Figure 5.6. 1-6. Logistic ANOVAs were 
conducted on the number of pilots who identified each weather information source as 
primary in the Wallops leg of flight. Logistic ANOVA Likelihood Ratio tests indicate 
that the type of weather display did not influence the likelihood that pilots would identify 
ATIS, PIREP, METAR, or a combination (i.e., Combo) of weather sources as primary in 
the Wallops leg of flight. Logistic ANOVA Likelihood Ratio tests did indicate that type 
of display influenced the probability that pilots would identify Pre -Flight Briefing 
(X 2 (2,40)= 9.22, p <.01), ATC (x 2 (2,40)= 6.06, p <.05), FSS/FW ( X 2 (2,40)= 6.17, p <.05) 
and Wx Display (x 2 (2,40)= 30.48, p <.001) as primary in the Wallops leg of flight. The 
results indicate that control pilots were significantly more likely to identify the Pre-Flight 
Briefing and FSS/FW as primary during the Wallops leg of flight than pilots with 
NEXRAD Image Looping display or NCWL display. Pilots with the NEXRAD Image 
Looping display were significantly less likely to identify ATC as a primary source of 
weather information than pilots with the NCWF display or control pilots. Also, control 
pilots did not identify Wx Displays as primary because they did not have access to this 
information. When compared to pilots with the NEXRAD Image Looping display, pilots 
with the NCWF display were less likely to identify Wx Display as a primary weather 
information source during the Wallops leg of flight. 
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Figure 5.6.1 -6. Pilots’ Self-Assessment of the Primary Source of 
Weather Information Used on Wallops Leg 


The number of pilots who identified each of the weather information sources available at 
Wallops as secondary is plotted by display type in Figure 5.6. 1-7. Logistic ANOVAs 
were conducted on the number of pilots who identified each weather information source 
as secondary in the Wallops leg of flight. Logistic ANOVA Likelihood Ratio tests 
indicated that type of display influenced the probability that pilots would identify Wx 
Display (x 2 (2,39)= 9.78, p < .01) as secondary in the Wallops leg of flight. Control pilots 
did not identify Wx Displays as a secondary source of weather information because they 
did not have access to this information. 
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Figure 5.6.1 -7. Pilots’ Self-Assessment of the Secondary Source of 
Weather Information Used On Wallops Leg 
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The observers recorded several workload considerations for both the Richmond leg and 
Wallops leg of the flight (see Appendix I, Observer Form). The workload considerations 
included communication lapses/errors, judgment errors, procedural errors, and flight-path 
deviations. The number of pilots that committed each type of error is plotted by display 
type in Figure 5.6. 1-8. Logistic ANOVAs were conducted on the number of pilots who 
committed errors during the Richmond leg of the flight. Logistic ANOVA Likelihood 
Ratio tests indicate that type of display had no significant influence on the likelihood that 
pilots would commit communication lapses/errors, judgment errors, procedural errors, or 
flight-path deviations in the Richmond leg of the flight. 
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Figure 5.6.1 -8. Pilot Workload Considerations on Richmond 
Leg as Recorded by Observers 


The number of pilots that committed communication lapses/errors, judgment errors, 
procedural errors, and flight-path deviations in the Wallops leg is plotted by display type 
in Figure 5.6. 1-9. Logistic ANOVAs were conducted on the number of pilots who 
committed errors in the Wallops leg of the flight. Logistic ANOVA Likelihood Ratio 
tests indicate that type of display had no significant influence on the likelihood that pilots 
would commit communication lapses/errors, judgment errors, procedural errors, or flight- 
path deviations on the Wallops leg of the flight. 


77 


Workload Considerations Wallops 


Comm. 

Lapses/Errors 

Judgment 

*2 Errors 

o 

lu Procedural 

Errors 

Flight-Path 
Deviations 

0 2 4 6 8 

# Pilots Committing Errors 


H 




□ NCWF 
■ Looping 

□ Control 

- 




1 




Figure 5.6.1 -9. Pilot Workload Considerations on Wallops 
Leg as Recorded by Observers 

5.7 Pilot Flight Control Actions, Navigation, and Communications 

The number of pilot induced control actions was extracted from the data recorded by the 
simulator. The measures of pilot induced control actions included the number of heading 
changes, airspeed changes and altitude changes. The number of pilot induced control 
actions is plotted by type of display in Figure 5.7-1. The control action variables were 
analyzed using MANOVA. The MANOVA and follow-up univariate ANOVA analyses 
found no significant associations between type of display and the number of pilot induced 
control actions. 
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Number of Significant Pilot Induced Control 
Actions by Display Type 
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Figure 5.7-1. Number of Pilot Induced Control Actions during Mission 


The number of radio frequency changes was extracted from the data recorded by the 
simulator. The number of frequency changes was recorded for both of the navigation 
radios (NAV 1 and NAV2) as well as the communication radio (COMM). The number of 
frequency changes for each radio is plotted by type of display in Figure 5.7-2. Univariate 
ANOVAs indicate there were no significant associations between type of display and 
number of frequency changes for any of the radios. 
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Figure 5.7-2. Number of Radio Frequency Changes (Communication 
and Navigation Radios) During Mission 
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5.8 Pilot Interaction with Autopilot 

The number of times the autopilot was engaged/disengaged, the number of times the 
heading hold function of the autopilot was engaged/disengaged, and the number of times 
the altitude hold function of the autopilot was engaged/disengaged were measures used to 
determine the degree of pilot interaction with the autopilot. The autopilot interaction 
measures are plotted by type of display in Figure 5.8-1. The autopilot interaction 
variables were analyzed using MANOVA. The MANOVA and follow-up univariate 
ANOVA analyses found no significant associations between type of display and the 
number of interactions with the autopilot. As Figure 5.8-1 illustrates, however, the pattern 
of results is quite consistent across the autopilot interaction measures. Pilots with the 
NEXRAD Image Looping display had the most interactions with all autopilot functions. 


Interactions with Autopilot 



□ NCWF 
■ Looping 

□ Control 


Number of Interactions 


Figure 5.8-1. Pilot Interactions with Autopilot During Mission 


The pilots had an autopilot available during the experiment and were trained on its use 
during the training session. During the pre -mission briefing, the pilots were instructed to 
use the autopilot if they felt it necessary to do so, but there was neither any requirement 
nor penalty in its use. Figure 5.8-2 illustrates the extent to which the pilots used the 
autopilot in the experiment. 
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Figure 5.8-2. Percent of Flight Time Autopilot Used During Mission 

Even though the autopilot was used for an average of 69% percent of the time in flight 
across all the participant pilots, some pilots were too busy to effectively integrate the use 
of the weather display into their procedures. Others were able to effectively use only one 
or two functions of the display. All of the pilots in both groups stated the autopilot was 
either essential to the safe accomplishment of the flight or substantially reduced the 
workload of flying in instrument conditions. Nearly all of the pilots stated that in 
reducing their workload, the autopilot made it possible for them to make more effective 
use of the weather information display. 

The average proportion of time the autopilot was engaged during the flight is plotted by 
type of display in Figure 5.8-3. Univariate ANOVA indicates that type of display did not 
significantly influence the proportion of time the pilots had the autopilot engaged, 
F(2,45)=.871, p<.43. As Figure 5.8-3 illustrates, however, the control pilots engaged the 
autopilot the lowest proportion of the time compared to pilots flying with weather 
displays. Thus, the results suggest that pilots flying with a weather display may have been 
experiencing higher workload than control pilots, which led them to engage the autopilot 
a higher proportion of the time. 
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Figure 5.8-3. Proportion of Flight Autopilot Engaged by Type of Display 

Most of the pilots said they felt comfortable with the autopilot and relied on its use to 
reduce the workload. The autopilot was, on average, in use for 69% percent of the flight 
(for all pilots). Many of the pilots stated that without the autopilot, they would have 
succumbed to an early termination of the flight. Because the RTI Simulation Facility is 
not a full-motion/full-reactive type of simulator, the lack of a peripheral visual system, 
motion base and reactive flight controls more than likely contributed to the extensive use 
of the autopilot. 
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6 Conclusions 

The purpose of the current experiment (CWE III) was to determine the effects of the use 
of NEXRAD looping and the use of the NCWF product on pilot workload and decision 
making. These two features were implemented as additional capabilities in a cockpit 
weather display already providing NEXRAD mosaic images overlaid on a moving map 
display. 

6.1 Confirmation of Experiment Design 

The elements of the experiment design were substantiated. These elements included the 
selection method for subject pilots, the prototype cockpit weather information display 
system design, the experiment scenario, the simulator fidelity (cockpit instrumentation, 
out-the-window scene generation, ATC communication environment, etc.), the use and 
recording of actual weather, the pre-flight training, the adequacy of the output data 
observed and recorded, the content and technique for the post-flight debriefing, and 
corroboration of the qualitative expert assessments with quantitative results. 

6.2 Support for Experimental Hypotheses 

Experimental Hypothesis #1 - Pilots using a NEXRAD image looping display will 
make significantly better decisions with respect to convective weather than will pilots 
without a weather information display. 

The experimental hypothesis that the NEXRAD mosaic image looping feature would 
improve pilot decision making was not supported by the results of the experiment. 

The results indicate that the NEXRAD mosaic image looping feature did not improve 
pilot decision making over pilots with no weather display in either the temporal decision 
at the Richmond Airport or the spatial decision enroute to the Wallops Island Facility. 

One possible explanation is that NEXRAD image looping may be difficult to use to 
predict storm movement. NEXRAD image looping provides a history of storm-cell 
movement, but no storm-movement forecast. Thus, the pilots must extrapolate the future 
movement of the hazardous weather themselves from the NEXRAD history, which could 
be a difficult task to perform when in flight with limited training. There is also 
insufficient information to be gained from the NEXRAD image history, at least as 
implemented in the NEXRAD mosaic image looping display used in the experiment, for 
the pilot to understand the speed of the movement of the hazardous weather or the rate at 
which the hazardous weather is increasing or decreasing in intensity. 

A significant part of the explanation is to be found in the observation during the 
experiment that many of the subject pilots overused the NEXRAD display, placing too 
much confidence in their interpretation of the information provided by the display. Rather 
than using their increased understanding of their situation gained from the display to 
complement their interaction with ATC and other conventional sources of weather 
information, they would proceed with a still incomplete understanding of the situation 
without seeking the additional information needed to make a good decision from the 
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other sources of information available. Resolution of this problem in the use of the 
NEXRAD display almost certainly includes better training. Further research is needed to 
determine the minimum level of training required. 

Experiment Hypothesis #2 - Pilots using a NEXRAD image looping display will 
experience significantly higher workload than will pilots without a weather 
information display. 

The experimental hypothesis that NEXRAD image looping would lead to increases 
in pilot workload was not supported by the results of the experiment. While a 
majority of the pilots flying with the NEXRAD mosaic image looping feature stated that 
using the weather information display increased their workload, many said in the next 
breath that the usefulness of the information gained from the display offset this increase 
in workload by reducing their anxiety and their need to find information from other 
sources . The observers in the study did not rate the workload of pilots having the 
NEXRAD image looping display to be significantly higher than pilots without a weather 
display. The level of interaction of the pilots with other sources of weather information, 
and with the autopilot also indicates that pilots with the NEXRAD image looping display 
did not experience significantly higher workload than control pilots. It is important to 
note, however, that the majority of pilots with NEXRAD image looping rarely used the 
looping feature according to the observers. Engaging the looping feature may have 
substantially increased pilot workload for those subject pilots who did so. Several pilots 
in the structured interview portion of the experiment stated that they did not have enough 
time to engage and interpret the NEXRAD image looping feature during the flight. The 
few pilots who did use the NEXRAD image looping feature extensively reported that 
while they felt the feature improved their understanding of the movement of the 
hazardous weather, they felt the time it took to use the feature increased their workload. 
Several pilots who used the NEXRAD image looping feature suggested that the number 
of keystrokes required to make the looping function available should be reduced. They 
suggested that only one keystroke be required to activate the image looping function. 

Experimental Hypothesis #3 - Pilots using the NCWF product display will make 
significantly better decisions with respect to convective weather than will pilots 
without a weather information display. 

The experimental hypothesis that pilots with the NCWF display would make better 
decisions concerning hazardous weather was not supported by the results of the 
experiment. Pilots with the NCWF display did not make better decisions than control 
pilots in the spatial decision at the Richmond Airport. Nor did pilots with the NCWF 
display make significantly better decisions than control pilots in the temporal decision 
enroute to Wallops Island. 

While nearly all pilots reported that the display greatly enhanced their understanding of 
the general location of the hazardous weather conditions relative to their own position, 
the delay in transmission of the NEXRAD mosaic image sometimes led to uncertainty 
and confusion in its use as a basis for decision making. Other aspects of the current 
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NCWF product incorporated in the display, such as the deletion of all precipitation radar 
returns below 17,000 feet and the inability of the product to correctly display hazardous 
conditions in their early development and as they began to deteriorate, increased lack of 
confidence in the display. 

A significant part of the explanation is to be found in the observation during the 
experiment that many of the subject pilots overused the NCWF display, placing too much 
confidence in their interpretation of the information provided by the display. Rather than 
using their increased understanding of their situation gained from the display to 
complement their interaction with ATC, they would proceed with a still incomplete 
understanding of the situation without seeking the additional information needed for a 
good decision from the other sources of information available. Resolution of this 
problem in the use of the NEXRAD display almost certainly includes better training. 
Further research is needed to determine the minimum level of training required. 

Another part of the explanation may lay in the complexity of the display. Several pilots 
reported they felt the display to be too cluttered. While most cockpit weather displays 
now in or about to reach the market have better resolution than the experimental display 
used in the experiment, it is likely that better resolution alone would not offset the 
difficulty in interpretation due to the clutter caused by the large amount of information 
displayed. Many of the pilots reported that while they understood the key features 
available in the display, significantly more training would be required to enable them to 
use the display more effectively. 

Experimental Hypothesis #4 - Pilots using the NCWF product display will 
experience significantly higher workload than will pilots without a 
weather information display. 

The experimental hypothesis that the NCWF display would lead to increases in pilot 
workload was not supported by the statistical data. Pilots with the NCWF display 
were not observed to experience significantly higher workload than pilots without a 
weather display. Analysis of the TLX questionnaire results also showed no significant 
difference in workload between the pilots using the NCWF display and pilots in the 
control group, although the overall pattern of TLX results suggests that pilots with the 
NCWF display generally rated themselves as experiencing higher workload than did 
pilots in the control group. The level of interaction of the pilots with other sources of 
weather information and with the autopilot also indicates that the pilots flying with the 
NCWF display did not experience a significantly higher workload than control pilots. 

There are several explanations why the NCWF display may have increased the pilots’ 
self-rating of their workload when compared to that of the pilots in the control group. The 
NCWF display provided a lot of information concerning hazardous weather that needed 
to be considered by the pilots (e.g., forecast projection, movement vector, precipitation 
tops, and NEXRAD radar), which could have contributed to the workload of the pilots. 
Also, because the NCWF display provides more information, it also has more clutter 
which might have made it difficult for the pilots to interact with the weather display and 
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contributed to their workload. In fact, several pilots specifically mentioned, during the 
structured interview, that the resolution of the weather display was poor and that clutter 
was somewhat of a problem. 

6.3 Comparison of Usefulness of NCWF Display to NEXRAD Image 
Looping Display 

The two cockpit weather displays used in the experiment, the NCWF Display and the 
NEXRAD Image Looping Display, were compared in terms of their contribution to the 
quality of the pilots’ decision making, their effect on pilot workload, their effect on pilot 
procedures, and their effect on the pilots’ situation awareness. 

6.3.1 Effect of Display Type on Decision Quality 

The subject pilots as a group made significantly poorer quality decisions in avoiding 
hazardous weather conditions during the approach to the Richmond Airport than they did 
while en route to Wallops Island. The pilots with weather displays did not make 
significantly higher-quality decisions on either leg than did the pilots without weather 
displays. And the effect of display type on the quality of the decisions made at Richmond 
and enroute to Wallops Island was not statistically significant. 

While the differences were not statistically significant, there were interesting trends in the 
differences in decision quality between the pilots having the NCWF display and those 
having the NEXRAD image looping display. Pilots with the NCWF display made the 
highest quality decisions on the Richmond leg when compared to quality of the decisions 
made by the pilots with the NEXRAD image looping display and those without a weather 
display, but they made the poorest quality decisions on the Wallops Island leg when 
compared to the other two pilot groups. Pilots with the NEXRAD image looping display 
made the highest quality decisions on the Wallops Island leg, but the poorest quality 
decisions at Richmond compared to the other pilot groups. These trends were found in 
the observers’ reports of the quality of the pilots’ decisions, and in the analysis of the 
minimum distance to the hazardous weather conditions experienced by the pilots. 

Of the two decisions, the relatively quickly changing weather conditions associated with 
the Richmond leg decision made this the more difficult of the two decisions for the pilots 
in all three groups (control, NEXRAD looping display, NCWF display) to deal with. 
Hence the poorer performance of all three pilot groups in dealing with this decision. 

The essence of the decision to be made on the Richmond leg of the scenario was to 
recognize that the hazardous weather conditions were more rapidly approaching the 
Richmond airport than had been forecast, to determine the location and severity of the 
hazard, to determine the direction and speed of movement of the hazardous conditions, to 
determine how close the pilot was willing to come to the hazardous conditions, to 
determine whether or not the approach could be continued into the airport without flying 
within the minimum safe avoidance distance of the storm, and if the decision was made 
not to continue the approach to then make the decision of where in the approach to 
change course. Of the two versions of the NEXRAD display, certain elements of the 
information unique to the NCWF display (storm movement vector, storm speed, and one- 
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hour prediction depiction) were probably more helpful in dealing with this complex 
decision than was the information unique to the NEXRAD image looping display 
(historical storm movement). 

The essence of the decision enroute to Wallops Island was 1) to recognize the location, 
dimensions, and severity of the hazardous weather conditions, 2) to determine the 
direction and speed of movement of the hazardous weather, and 3) once it was realized 
the hazardous area was fairly static and localized to an area over the Chesapeake Bay, to 
decide how best to circumnavigate the hazardous area. Since the weather conditions 
associated with the Wallops Island leg decision were somewhat static, the NEXRAD 
image looping function probably provided an easier and perhaps more intuitive 
assessment of the lack of storm movement while the two displays were equally effective 
in supporting the necessary interpretation of the dimensions of the hazardous conditions 
(localized over the Chesapeake Bay). 

6.3.2 Effect of Display Type On Pilot Workload 

While the subject pilots as a group were observed to have experienced significantly 
higher workload during the approach to the Richmond Airport than they did while en 
route to Wallops Island, the pilots with weather displays were not observed to have 
experienced significantly higher workload than pilots without weather displays. 
Likewise, the analysis of the pilots’ self-ratings of workload found no significant 
difference in workload between display type (although there is a suggestion that pilots 
with the NCWF display generally rated themselves as experiencing a higher workload 
than did pilots with the NEXRAD image display). The analysis of the observers’ reports 
of workload considerations such as pilots’ communication lapses and errors, judgment 
errors, or flight path deviations also found that type of display had no significant 
influence. 


6.3.3 Effect of Display Type On Pilot Procedures 

Analysis of data extracted from the simulator indicates that the type of display had no 
significant effect on the number of pilot control actions (heading changes, airspeed 
changes, altitude changes, communication radio changes, navigation radio changes) 
during the mission. Analysis of the interaction of the pilot with the autopilot found no 
significant influence of display type on either the number of interactions with the various 
autopilot functions or on the proportion of the flight in which the autopilot was used. 

The display type had no significant effect on the extent to which the subject pilots having 
the two different displays used other sources of weather information (ATC, 
ASOS/AWOS, FSS, etc), or on the extent to which they felt the need to crosscheck or 
verify their conclusions drawn from the weather display with other conventional weather 
data sources, or in their perception of the need to crosscheck their inferences drawn from 
the weather display, or on the extent of their agreement that they were aware of the age of 
the information depicted on their weather display. 
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There was a significant difference in the extent to which the pilots of the two display 
groups perceived that they used the weather display to help them navigate. A relatively 
large number of pilots with the NEXRAD image looping display reported that they used 
the weather display to help them in their navigation procedures to avoid the convective 
weather conditions. The pilots having the NCWF display were evenly divided in the 
perception as to whether they used their weather display to help them navigate. The 
difference in the perceptions of the pilots of the two display groups as to their use of their 
weather display to help their navigation did not show up, however, in their performance. 

6.3.4 Effect of Display Type On Pilot Situation Awareness 

Nearly every subject pilot in the control group stated they could have better performed 
the mission had they been equipped with an onboard weather display of some type. And 
nearly every subject pilot in the two display groups stated they felt the weather display 
greatly improved their awareness of their situation with respect to the location and 
movement of the convective weather conditions along the route. The pilots of both 
display groups stated that the weather display increased their situational awareness during 
the decision-making process. 

There were no significant differences in the responses of the pilots having the two 
different displays to the questions regarding their perception of the proximity of 
convective weather conditions to the Richmond Airport and across their path enroute to 
Wallops Island, or to the questions regarding the degree to which they knew their relative 
position to storms when relying on their weather display, or to the questions regarding 
their confidence that they had adequate sources of information to make confident 
decisions. 

There were no significant differences in the responses of the pilots having the two 
different displays to the questions regarding their understanding of the latency of the 
information provided by their weather display. 

6.4 Effectiveness of Modifications to the Basic NEXRAD Display 

The graphical weather depiction age indicator, added to the lower left comer of both of 
the NEXRAD displays so that pilots could, at a glance, determine the age of the weather 
information, was found to have substantially reduced the mental effort required to 
calculate the age of the weather information. 

Translation of the text METARs from the ICAO format to English text substantially 
decreased pilot workload for most of the subject pilots, enabling them to more effectively 
use the text METAR. An option should be provided to permit pilots to choose between 
the ICAO format and the translated format. 

A depiction of flight path, including key waypoints and their identifiers, is essential for 
effective use of a NEXRAD mosaic image display. 

Provision of a 25 nm range-ring on the 10 mile and 25 mile scales was found to 
substantially improve the pilots’ determination of their proximity to hazardous weather. 
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Also, making the 25 nm range-ring invariant across display scale selection (i.e., range 
modes) substantially reduced mode errors associated with the display scale selection. 

6.5 Use of Autopilot to Complement Weather Information Display 

This study was not designed a priori to measure the effects of the use of an autopilot on a 
pilot’s ability to safely and effectively use the cockpit weather information display. In 
this experiment, however, the subject pilots equipped with the weather information 
display and the observers were nearly unanimous in their assessment that the autopilot 
contributed significantly to the safe and effective use of the display. The observers 
reported a clear correlation between the extent of the subject pilot’s use of the weather 
information display and their use of the autopilot. 

Even though the autopilot was used for an average of 69% percent of the time in flight 
across all the participant pilots, some pilots were too busy to effectively integrate the use 
of the weather display into their procedures. Others were able to effectively use only one 
or two functions of the display. All of the pilots in both groups stated the autopilot was 
either essential to the safe accomplishment of the flight or substantially reduced the 
workload of flying in instrument conditions. In many cases the subject pilots reported 
they would not have continued with the flight had they not had the use of the autopilot. 
Nearly all of the pilots stated that in reducing their workload, the autopilot made it 
possible for them to make more effective use of the weather information display. 
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7 Recommendations 


The following recommendations are based on the findings of this study, loosely 
formatted for possible incorporation in the FAA Aeronautical Information Manual 
(AIM), draft FAA Advisory Circular No: 00-FIS, titled “Use of Cockpit Displays of 
Digital Weather and Operational Information,” and draft FAA Advisory Circular No: 20- 
FIS, titled “Safety and Interoperability Requirements for FIS Equipment”. The 
recommendations pertaining to the AIM are limited to information that does not duplicate 
information already provided in the latest edition of the AIM (February 1, 2002). 

Additional recommendations are provided for the consideration of the FISDL display 
system manufacturers. 

7.1 Aeronautical Information Manual and Advisory Circular 
Recommendations 

The depiction of weather information, including NEXRAD and METAR products, will 
be delayed due to the time required for the collection and distribution of vast amounts of 
weather information available. 

The time required to produce the NEXRAD mosaic display includes a six-minute cycle 
for the individual NEXRAD radars to scan and observe the data. An additional interval is 
required for the automated processing of the NEXRAD data necessary to merge all the 
individual NEXRAD radar images into one national mosaic before the NEXRAD 
national mosaic is available from which to create the FISDL cockpit images for 
transmission. 

METAR observations are only produced at the top of the hour. The hourly METAR 
observation remains as the "official" observation for the airport throughout the hour and 
is included in the airport ATIS. During approach the pilot will be provided the direct 
readout wind and altimeter information by the tower controller. Also, pilots can obtain 
the aural report of the latest minute ASOS observation while in radio range of the airport 
ASOS. During dynamic, changing weather conditions, SPECI observations are issued 
and are included in new ATIS reports, but they are unscheduled and are thus 
unpredictable in terms of knowing or anticipating when they should be available. Also, 
TAF forecasts are issued four times per day at scheduled intervals and remain valid until 
amended or superceded by the next issued TAF. TAF AMEND, like SPECI 
observations, are unscheduled and are thus also unpredictable in terms of when they 
should be available. The availability of SIGMET, AIRMET, PIREP and AWW reports is 
similarly unpredictable in that they are primarily event driven and issued (or 
amended) when weather conditions dictate. 

Another delay introduced into all the FISDL products is a product of the FISDL 
broadcast transmission cycle. The communication architecture of the FISDL broadcast 
will determine the magnitude of that delay for any specific FISDL product. For example, 
the FISDL Service Provider may decide to place a priority on transmitting NEXRAD 
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products and thus "interrupt" any text transmissions when a new NEXRAD product is 
received. 

It is essential that the pilot become fully proficient in determining and maintaining a 
comprehensive awareness of the age of each of the FISDL display weather information 
products so as to be able to effectively and accurately integrate this information 
(NEXRAD image time stamps, METAR text time data, etc.) with the information 
gathered from the other sources. 

Pilots should be fully aware that the FISDL display does not contain sufficient 
information to support navigation, and it should not be used as a replacement for any 
aspect of approved navigation procedures and equipment. While the FISDL display can 
increase the pilot’s situational awareness, particularly with respect to weather conditions, 
the display cannot be successfully used to determine headings, direction, or distances 
with the accuracies and reliability that are required for navigation. 

The mental activity required to use the FISDL display can increase the pilot’s workload 
in instrument conditions for some pilots. An autopilot can offset this workload increase, 
freeing up the mental processes to support more effective use of the display. Some pilots 
have reported that an autopilot is essential to their effective use of the FISDL display. 

7.2 Weather Display Manufacturer Recommendations 

7.2.1 Consider Providing Flight Path Information 

Both subjective and objective measures have found in previous experiments in which 
flight path information was not displayed on the weather information display that most of 
the pilots had difficulty determining their position in relation to the weather and their 
distance to the convective weather activity. With the proliferation of moving map 
displays in modem cockpits, pilots are used to seeing ownship symbology, which they 
use to determine their position, and flight path information, which they use to determine 
proximity of hazardous convective weather information to their future flight path. 

The benefits of ownship symbology and flight path depiction appear to outweigh the 
concerns associated with the display of real-time position information (ownship ground 
track) and old information (7 minute old NEXRAD) on the same display, During the 
post- flight briefings with the subject pilots, most commented that they realized the 
NEXRAD images were not real time and would take the staleness into account when 
comparing their future position and flight path in relation to the weather depiction. 

7.2.2 Provide Direction and Rate of Hazardous Weather Motion 

Many of the pilots in previous studies had difficulty determining the movement of the 
convective weather and asked for either a looping capability (playback of preceding 
images) or vector arrows showing speed and direction (similar to the National Weather 
Service radar depiction charts). In this study, the pilots stated their situation awareness 
with respect to the hazard of convective weather activity in their vicinity was 
significantly increased. 
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7.2.3 Consider Providing A Range Ring to Assist Distance Determination 
and Avoid Mode Confusion 

Many of the pilots in previous experiments made poor estimations of the distance 
between the aircraft and the convective weather. This misperception was a significant 
contributor to the inability of many of the pilots to effectively use the FISDL display. In 
this study the pilots’ performance and reduced workload argue persuasively for the 
provision of a range ring around the ownship symbol to assist in more accurate estimation 
of distances and to assist in minimizing pilot confusion as to the display range scale that 
has been selected. 

7.2.4 Provide Intuitive NEXRAD Image Age Information 

In this study the pilots’ performance and reduced workload argue persuasively for the 
provision of an intuitive indication of the age of the NEXRAD mosaic image. 

7.2.5 Provide METAR Code Translation As A Pilot Option 

In this study a translation of the METARs into English text was provided when the pilots 
selected the text METAR feature on their cockpit weather display. In this study, nearly all 
of the pilots used the METAR text reports to one degree or another - a major 
improvement over the very limited use of these reports in previous studies in which the 
reports were provided in the ICAO format. A majority of the general aviation pilots’ who 
participated continued to comment, as they have commented in previous studies, that 
when presented in the ICAO METAR format the METAR reports are difficult to 
interpret, and take too much time to interpret. Only those few general aviation pilots who 
use the ICAO METAR format regularly reported that they were more comfortable with 
the ICAO format and felt that it substantially reduced the time required to assimilate the 
information provided. 

Additionally, consideration should be given to the provision (via data link) of direct 
readouts of current controlling conditions to the pilot. 

7.2.6 Consider Providing the NEXRAD Image Looping Feature As an 
Option 

In this study, the NEXRAD image looping feature provided a relatively intuitive and 
useful means of enabling a pilot to determine the direction of movement of convective 
weather activity. Manufacturers of data link NEXRAD weather displays could consider 
providing this feature as an optional feature of their product. 

7.2.7 Develop and Provide Comprehensive User Training for FISDL 
Display Customers 

In this study, the display of NEXRAD mosaic images substantially increased the pilots’ 
awareness of the general location of convective weather in their vicinity. The compelling 
nature of the display of these images, however, caused some pilots to depend too heavily 
on the display for their information regarding hazardous convective weather condition. 

As a result, they failed to obtain other essential and corroborating information from other 
available sources and proceeded to fly into what would have been extremely hazardous 
conditions had they not been flying in a flight simulator. Substantial training in the use 
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of the FISDL display system is required to help pilots understand the limitations of the 
FISDL display and its data, to reduce the workload otherwise required to access and 
interpret weather information, and to enable the pilot to fully exploit the potential safety 
contributions of the display. 

7.2.8 Emphasize Need for Autopilot for Safe and Effective Use of FISDL 
Display 

In this study, nearly every pilot emphatically stated that safe and effective use of the data 
link cockpit weather display can only be undertaken with the assistance of a capable 
autopilot. Their performance underscored the importance of the use of the autopilot as a 
means to offset the additional workload incurred in using the cockpit weather display. 

7.3 Recommendations for Further Research 

Proposed research topics fall into three broad categories: evaluation of specific issues of 
interest to the FAA, weather information integration and display design enhancements, 
and development of a training system for aircraft cockpit weather information 
management. 

7.3.1 Conduct Evaluations of Specific FISDL Issues to Support Standards 

This research topic is motivated by the need for standards and design guidelines to assure 
safety. Issues to be addressed include 1) how to minimize the effort required to learn to 
use the display, 2) how to facilitate interpretation of the display, 3) how to minimize the 
likelihood that the displayed weather information can be misused, leading to poor 
navigation decisions, and 4) how to efficiently obtain and coordinate the data necessary 
for incorporation in appropriate standards. 

7.3.2 Develop Concepts for Information Integration and Display Design 
Enhancements 

A variety of weather related products and onboard sensor derived data are currently 
available or pending implementation in aircraft, e.g., NEXRAD images, lightning data, 
data linked icing warnings, convective weather turbulence, etc. Display concepts are 
needed which integrate the available data into more useful representations of spatial and 
temporal information incorporating the lessons learned to date by the RTI/NASA team 
and other organizations working in this area. The cognitive skills required of the pilot to 
integrate weather data from many sources would be greatly reduced, as would the 
comprehensive weather data interpretation training currently required. 

Further investigation should be undertaken to determine the extent to which the NCWF 
product, or at least selected features of this capability as it now exists, might be 
incorporated in cockpit weather information displays. 

7.3.3 Develop Training Curriculum for Weather Information Displays 

A training curriculum should be developed to support the implementation and proper use 
of weather displays in the cockpit. The curriculum needs to include appropriate manuals 
and modem interactive multi-media training techniques that would highlight common 
mistakes and improper usage of the weather display information, and develop and 
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reinforce appropriate operational procedures for the use of weather display systems. 
Accompanying experiments should be undertaken to evaluate the efficacy of the training 
curriculum. 
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Appendix A. Pre-Flight Weather Briefing 


Pre-Flight Weather Briefing 

As part of the mission preflight briefing materials, each pilot was given a paper copy of a 
standard weather briefing that would have been received by a call to a Flight Service 
Station telephone briefer. Both the teletype coded reports were given as well as an 
English translation. 


Adverse Conditions: 


AIRMET (WA) TANGO FOR TURB VALID UNTIL 272100Z 

AIRMET TURB...MD VA NC 

FROM EMI TO SBY TO RDU TO PSK TO EMI 

AFT 18Z OCNL MOD TURB BLW 060 DUE TO INCRG SWLY FLOW AHD OF 
CDFNT. CONDS SPRDG EWD AND CONTG BYD 21Z THRU 03Z. 


AIRMET (WA) TANGO for turbulence valid until twenty-one hundred universal 
coordinated time for Maryland, Virginia, and North Carolina. 

From Westminster (EMI), Virginia to Salisbury (SBY), Maryland, to Raleigh-Durham 
(RDU), North Carolina to Pulaski (PSK), Virginia to Westminster (EMI), Virginia. 
After one, eight, zero, zero, universal coordinated time, occasional moderate turbulence 
below six thousand feet due to increasing southwesterly flow ahead of cold front. 
Conditions spreading eastward and continuing beyond twenty-one hundred universal 
coordinated time, and through zero, three, zero, zero universal coordinated time. 


Synopsis: 

At one, seven, zero, zero universal coordinate time, a Cold Front extending from 
southwest Pennsylvania along the Appalachians through Central West Virginia, Western 
Virginia, and Eastern Tennessee, northwest Georgia and Central Alabama will continue 
to move Eastward. 

A warm front extending from southwest Pennsylvania Eastward to Atlantic City, NJ. will 
continue to move Northeastward, and a Trough of Low Pressure extending from 
northwest West Virginia southward into Central South Carolina will continue moving 
Eastward. 
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Current Conditions: 


PHF SA 1800Z M 8 BKN 07 14/12/0910/992 

Newport News, Williamsburg International Airport weather report, one, eight, zero, zero 
universal coordinated time. Measured ceiling eight hundred broken, visibility seven, 
temperature one, four, dew point one, two, wind zero niner, zero at ten, altimeter two, 
niner, niner, two. 

RIC SA 1800Z 50 SCT M70 BKN 05 14/12/3010/992 

Richmond International Airport weather report, one, eight, zero, zero universal 
coordinated time. Five thousand scattered, measured ceiling seven thousand broken, 
visibility 5, temperature one, four, dew point one, two, wind three, zero, zero at one zero, 
altimeter two, niner, niner, two. 

OFP SA 1747Z E50 BKN 150 OVC 10 18/12/2015/960 

Richmond, Hanover County Airport weather report, one, seven, four, seven universal 
coordinated time. Estimated ceiling five thousand broken, one, five thousand overcast, 
visibility one, zero, temperature one, eight, dew point one, two, wind two, zero, zero, at 
one, five, altimeter two, niner, six, zero. 

LKU SA 1750Z 20 SCT E40 BKN 100 OVC 10 17/12/2415G20/955 

Louisa County, Freeman Airport weather report, one, seven, five, zero universal 
coordinated time. Two thousand scattered, estimated ceiling four thousand broken, one, 
zero thousand overcast, visibility one, zero, temperature one, seven, dew point one, two, 
wind two, four, zero at one, five gusting two, zero, altimeter two, niner, five, five. 

WAL SA 1749Z CLR BLO 120 10 16/10/1810/969 

NASA, Wallops Airport weather report, one, seven, four, niner universal coordinated 
time. Clear of clouds below one, two thousand, visibility one, zero, temperature one, six, 
dew point one, zero, wind one, eight, zero at one, zero, altimeter two, niner, six, niner. 

MFV SA 1753Z CLR BLO 120 10 18/11/1806/968 

Accomack County Airport, Virginia weather report one, seven, five, three universal 
coordinated time. Clear of clouds below one, two thousand, visibility one, zero, 
temperature one, eight, dew point one, one, wind one, eight, zero at six, altimeter two, 
niner, six, eight. 

SBY SA 1750Z CLR BLO 120 10 16/10/1810/969 

Salisbury, Maryland weather report one, seven, five, zero universal coordinated time. 
Clear of clouds below one, two thousand, visibility one, zero, temperature one, six, dew 
point one, zero, wind one, eight zero at one, zero, altimeter two, niner, six, niner. 
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UA: /OV RIC150015 /TM 1720Z /FL 040/TP C180 /SK SCT150 /TB LGT 

Pilot report one-five miles southeast of Richmond, Virginia. At one, seven, two, zero 
universal coordinated time. At four thousand feet, a Cessna one, eighty reported in clouds 
with light turbulence. 

UA: /OV SBY /TM 1715Z /FL 030 /TP MO20 /SK SCT150 /TB NEG 

Pilot report over Salisbury, Maryland at one, seven, one, five universal coordinated time. 
At three thousand feet, a Mooney reported clouds at one five thousand scattered, and 
negative turbulence. 

UA: /OV RIC045025 /TM 1710Z /FL 040 /TP C172 /TB LGT-MOD 

Pilot report two-five miles northeast of Richmond, Virginia at one, seven, one, zero 
universal coordinated time. At four thousand feet, a Cessna one, seven, two reported light 
to moderate turbulence. 

Satellite Imagery indicates several Cumulus clouds beginning to develop throughout 
central Virginia, including the Richmond area, over the past hour. 

Weather Radar at one, seven, one, zero universal coordinated time indicates scattered 
areas of light to moderate rain showers in Central Virginia, but no precipitation in the 
Eastern sections of the state. 


En-Route Forecast: 


TAF KPHF 271729Z 271818 16014G24KT P6SM SCT100 BKN200 BECMG 2022 
16017G27KT SCT060 OVC120 

FM0000 1618G25KT P6SM SCT030 OVC060 TEMPO 5SM -SHRA OVC030 
PROB40 0103 VRB20G40KT 2SM TSRA OVC020 CB 

Terminal area forecast for Newport News-Williamsburg International Airport. Valid from 
one seven, two nine, to one eight, one eight, universal coordinated time, wind one, six, 
zero at one, four gusting two, four, visibility unrestricted, scattered clouds at one, zero 
thousand, broken clouds at two, zero thousand. Conditions becoming between two, zero, 
zero universal coordinated time and two, two, zero, zero universal coordinated time, wind 
one, six, zero at one, seven gusting two, seven, scattered clouds at six thousand, overcast 
at one, two thousand until zero, zero, zero, zero universal coordinated time. 

Central and Eastern Virginia Area Forecast: 


271800Z SCT-BKN050 OVC120 TOP 200, OTLK VFR TSRA 

The area forecast for Central and Eastern Virginia after one, eight, zero, zero universal 
coordinated time: scattered to broken clouds at five thousand, overcast at one, two 
thousand, tops at two, zero thousand, outlook VFR with thunderstorms and rain. 
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TAF KRIC 271729Z 271818 18018G20KT P6SM SCT060 OVC120 BCMG2022 
OCNL -SHRA OVC030 PROB40 2302 VRB20G40KT 2SM TSRA OVC020CB 

Terminal area forecast for Richmond International Airport. Valid from one seven, two 
nine, to one eight, one eight, universal coordinated time, wind one, eight, zero at one, 
eight gusting two, zero, visibility unrestricted, scattered clouds at six thousand, overcast 
at one, two thousand. Conditions becoming between two, zero, zero, zero universal 
coordinated time and two, two, zero, zero universal coordinated time, occasional light 
rain showers, overcast at three thousand, with a chance of thunderstorms after two, three, 
zero, zero universal coordinated time. 

TAF KSBY 271729Z 18018 1820G30KT SCT100 BKN200 

Terminal area forecast for Salisbury, Maryland after one seven, two nine, universal 
coordinated time. Wind one, eight, zero at two, zero gusting three, zero, scattered clouds 
at one, zero thousand, broken clouds at two zero thousand, visibility unrestricted. 

Winds Aloft Forecast: 

030 060 090 

ORF 1920 2025+5 2130+2 
RIC 2020 2125+4 2130+1 

Winds aloft forecast for the Norfolk, and Richmond, Virginia areas after one, seven, zero, 
zero universal coordinated time. Norfolk at three thousand: wind one, niner, zero at two, 
zero. At six thousand: wind two, zero, zero at two, five, temperature plus five. At niner 
thousand: wind two, one, zero at three, zero, temperature plus two. Richmond at three 
thousand: wind two, zero, zero at two, zero. At six thousand: wind two, one zero at two, 
five, temperature plus four. At niner thousand: wind two, one, zero at three, zero, 
temperature plus one. 

NOTAMS: 

No Current NOTAMS Listed. 

ATC Delays: 

NONE 

ATC request PIREPS for turbulence or other conditions along your route of flight. 
Contact Flight Watch or Flight Service. Washington Flight Watch is available with En- 
route Flight Advisory Service to update your weather briefing on 122.0MHz. Leesburg 
Flight Service station is available on 122.2 MHz for weather briefings and other in-flight 
services. 
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Appendix B. Cockpit Research Facility Description 


1. Cockpit Research Facility (CRF) Description 

The Cockpit Research Facility (CRF) provides a means to evaluate both conventional and 
experimental cockpit displays and flight control configurations for single engine General 
Aviation (GA) aircraft. Experiments can be performed with any combination of 
conventional or experimental instruments/displays and controls in a fully immersive 
communications, navigation and surveillance (CNS) environment. Illustrated in Figure 
B-l, the CRF consists of three major components: 

• Rapid Prototype Simulator Cockpit - The cockpit mockup with controls, 
instruments and indicators. 

• Scenario Controller - The master control station used for scenario generation, 
selection, monitoring, fault injection and recording of flight progress. This 
station transfers all data between the controls and indicator in the cockpit and the 
simulation models in the Scenario Controller. Display of all current control 
positions, switch positions and instrument displays and Out- the- Window scene 
are also available. A Closed Circuit Television (CCTV) is used for monitoring 
and recording subject pilot’s actions and conversations with simulated Air Traffic 
Manager (ATM). 

• ATM Controller - An extension of the Scenario Controller, provides a plan view 
of the simulated aircraft’s progress and flight plan, similar to an ATC console, 
along with other traffic in the scenario. A display of the current pilot-selected 
COM frequencies is available so that the ATM controller can verify that the pilot 
is contacting ATM on the correct frequency before responding to an initial 
contact. 

A high-level diagram of the major system components is illustrated in Figure B-2. These 
major components are described in the following paragraphs. 
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Figure B-2. Simulated Flight (Rapid Prototype Simulator Cab) 




Figure B-3. Control Room (Scenario Controller) 



Figure B-4. ATM Station 
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Figure B-5. SGI Configuration - 
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Figure B-6. PC Configuration - Cockpit Research Facility System Block Diagram 
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2. Rapid Prototype Simulator Cockpit 

2.1 Overview 

The simulator cockpit is a two-seat cockpit mockup designed for single -pilot IFR 
operations. The basic ergonomic structure of the mock-up is patterned after the Global 
Aircraft GT-3 Trainer in terms of the relative placement and types of controls and 
instruments, instrument panel width and height, and seat placement. The pilot’s position 
is outfitted with complete controls including a yoke, rudder pedals, instruments, switches 
and indicators as described in the paragraphs below. A left- or right-handed side stick 
controller is also available. Virtually any flight control system can be implemented. The 
second seat is used for a test subject observer or non- flying copilot. The interior 
dimensions of the cockpit enclosure are similar to those of the Cessna 208 Caravan 
aircraft. 

The major features include: 

• Rapid Reconfiguration of the Instrument Panel - The most flexible aspect of the 
simulator cab is the ability to quickly change the types of instruments in the 
instrument panel. Instrument panels currently simulated include the round dial and 
separate PFD/MFD types. Various PFDs and MFDs are also supported and can be 
switched into or out of the simulator cab in minutes. Other PFD/MFD systems 
supported include Seagull, ARNAV 5200 / 2000 series , Archangel, Nav3D systems, 
UPSAT hardware, and Global/RTI International low-cost HUD configuration. 

• Rapid Reconfiguration of the Cockpit Geometry - The simulator cab design facilitates 
rapid changes in configuration. The center console can be moved in or out as 
required to approximate a cockpit geometry. The rudder pedal placement is adjustable 
through a wide range of distances from the pilot’s seat. The seats are adjustable in 
distance from the yoke / instrument panel. Further, the pilot’s seat and rudder pedals 
can be moved fore and aft as a single unit in the cab to accommodate most subject 
pilots and future control mechanisms. 

• Reconfigurable Flight Controls - The flight controls currently available include a 
control yoke with simulated cable / mechanical control surface actuation. An 
electronic side stick hand controller is also available. Both control systems include 
autopilot assistance and electric trim. 

• Reconfigurable Power Controls - The center console holds the throttle quadrant and 
radios. The throttle quadrant can be replaced with the traditional push rod 
configurations for throttle, mixture and carburetor heat, or a single lever power 
control and simulated Full Authority Digital Engine Control (FADEC). 

• Spoiler System - The aero model includes a spoiler system that is controlled by the 
throttle position which decreases lift, increases drag for investigation into higher 
approach speeds and steeper approach angles. This spoiler system can be used in 
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conjunction with a Head Up Display (HUD) system to monitor the effect of the 
spoiler deployment on the flight path. 

• Multiple Visual Scene Databases - The primary out-the-window view is currently 
provided by a projection system mounted in the ceiling directly over the pilot, and 
projects an image on a 10 foot wide screen approximately 12 feet from the pilot. 
Visual terrain scenery includes the Denver terminal area and the state of Virginia with 
six detailed airports with associated navigation aids. 

• Intercom and Video System - Both cockpit positions, Scenario Controller position 
and ATM position are outfitted with headsets and an intercom system that allows the 
pilot to communicate with the passenger and with simulated Air Traffic Manager. 

The passenger (experiment observer) can only communicate with the pilot. A 
separate hookup is available in the control room for two observers to be placed on the 
intercom system with communications with the Scenario Controller and the ATM. 

The Scenario controller can monitor all conversation. All audio is recorded on video 
tape along with a video image of the subject pilot’s actions. Playback of specific 
ATIS and AWOS stations are available when the pilot tunes to the desired frequency. 


2.2 Controls, Instruments and Indicators Description 

Multiple instrument panel configurations are available as illustrated in Figure B-7 
through B-l 1. A conventional “round dial” displays configuration or any combination of 
PFD/MFD/round dial and HUD displays are possible as shown. The controls, 
instruments and indicator configurations available are typical of those found in an IFR- 
equipped general aviation aircraft. 



Figure B-7a. Conventional Round Dial Instruments 
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Figure B-7b. Conventional Round Dial Instruments 



Figure B-8a. Conventional Round Dial Standard “T” and Electronic MFD 



Figure B-8b. Conventional Round Dial Standard “T” and Electronic MFD 
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Figure B-8c. Conventional Round Dial Standard “T” and Electronic MFD 




Figure B-9b. Conventional Electronic PFD (EFIS) and Electronic MFD 
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Figure B-IOa. 3-D PFD with Terrain and Traffic, and Electronic MFD 



Figure B-1 Ob. 3-D PFD with Terrain, Pathway , Traffic and 
Weather, and Electronic MFD 



Figure B-1 la. Electronic PFD, FMS CDU, MFD and HUD with 3-D Terrain, 
Pathway, Traffic, Special Use Airspace and Weather. 
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Figure B-llb. Electronic PFD and HUD with 3-D Pathway, 
Terrain, and Traffic 


The key characteristics of these configurations can be summarized as follows: 

• Round Dial Instrument Displays - All instrument panel round dial indicators are 
realistically rendered on flat panel liquid crystal displays (LCDs). The 14-inch 
diagonal LCDs provide enough display area to fully render the standard instrument 
“T” configuration with other supplemental indicators as well. A second 14-inch 
diagonal LCD provide display area for navigation instruments and engine parameter 
indicators. All instruments can be designed to recognized Federal Aviation 
Administration (FAA) Federal Aviation Regulations (FARs), Society of Automotive 
Engineers (SAE) Aerospace Standards (AS) and RTCA, Inc. performance 
specifications. Or alternately, an experimental instrument can be designed and 
displayed that has enhanced or unique characteristics. The instruments can be 
rendered anywhere on the two LCD displays, in accordance with the FARs or in 
experimental arrangements. 

• Electronic Primary Flight Displays (PFDs) and Multi-Function Displays (MFDs) and 
HUDs. The instrument panel can accommodate self-contained PFDs and MFDs or a 
combination of PFD/round dial, including a simulated HUD. A PFD, MFD or other 
type of display is provided with the required data elements (in accordance with the 
manufacturer’s Interface Control Document (ICD)) generated by the simulation aero 
and engine models. Three-Dimensional PFDs and HUDs are currently supported. 
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One 3-D implementation includes the display of traffic, 3-D pathway as well as 
terrain. Another implementation provides a 3-D pathway and flight director. 

Figure B-12 illustrates the full round dial CRF configuration. Table B-l contains the 
types of instruments available for rendering in the cockpit. Flight control instruments 
provide the pilot with the parameters of the simulated aircraft state vector. Besides a 
gyro stabilized directional compass, two VOR / DME displays complete with ILS Course 
Deviation Indicator bars can be used for navigation and approach to landing. 

The simulated radio stack in the center console includes COM, NAV radios, an ADF 
receiver, DME interrogator, transponder and autopilot controls. The radio stack is used to 
select the proper COM frequencies to obtain ATC clearances, select the required VOR / 
DME and ILS frequencies. Two Omni-Bearing Selector (OBS) knobs located on the 
simulated radio stack are used to control the VORs. Two additional knobs located on the 
radio stack are used to set the barometric pressure for the altimeter and the Directional 
Gyro (DG). These knobs are located on the console since actual instruments are not 
used. 

The control yoke provides the pilot with an electric trim button, push-to-talk switch for 
the intercom system, autopilot disconnect switch, chronometer for time calculations as 
well as a full range of control movement for controlling the flight path of the airplane. A 
hand controller is also available and can be located at the end of either arm rest. The 
hand controller contains electric trim and an intercom PTT switch. When configured 
with the yoke, activation of the electric trim button moves the yoke in or out to relieve the 
control forces. When using the hand controller, the electric trim button allows the pilot to 
return the stick to the centered position. The current trim position is displayed on a trim 
indicator in the instrument panel. 
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Figure B-12. CRF Round Dial Instrument Panel, Controls 
and Indicators Configuration 


Table B-1. Instruments and Indicators 


Flight Control 
Instruments 

Navigation / 
Communications 
Instruments 

Engine Monitoring 
Instruments and Control 

Indicators 

• Attitude 

• Airspeed 

• Altitude 

• Turn and Bank 

• Slip & Skip 

• Gyro Stabilized 
Direction 

• Vertical Speed 

• Autopilot 

• VOR / DME 
Display #1 with ILS 
Localizer & 
Glideslope 

• VOR / DME 
Display #2 with ILS 
Localizer & 
Glideslope 

• NAV Radio #1 

• NAV Radio #2 

• COM Radio #1 

• ADF 

• Transponder 

• Manifold Pressure (MAP) 

• Engine RPM 

• Fuel Quantity 

• Fuel Flow 

• Oil Pressure 

• Exhaust Gas Temperature 
(EGT) 

• Cylinder Head Temperature 
(CHT) 

• Throttle, Mixture, Prop 
Levers 

• Throttle, Mixture, Prop 
Push-rods 

• Single Lever Power Control 

• Trim Position 

• Flap Position 

• Gear Position 
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3. Visual System and Displays Description 

SGI Version: 

A Silicon Graphics Onyx 10000 is used to generate the instrument panel gages and the 
Out-the Window (OTW) scene for the pilot as depicted in Figure B-2. A rapid 
prototyping tool is used to develop and render the regulation-compliant gages in 
appearance and performance. Round dial instruments and AGATE PFD Highway-in-the- 
Sky (HITS) configurations are rendered on two 14-inch diagonal active matrix Liquid 
Crystal Displays (AMLCDs), each having an addressable resolution of 1024 pixels by 
768 lines. Any type of instrument face or display can be rendered, including new and 
novel displays. 

PC Version: 

Separate Evans and Sutherland simFusion 4000Q are used to generate the instrument 
panel gages and the Out-the Window (OTW) scene for the pilot as depicted in Figure B- 
3. A rapid prototyping tool (VAPS) is used to develop and render the regulation- 
compliant gages in appearance and performance. Round dial instruments and 3-D view 
PFD Highway- in- the-Sky (HITS) configurations are rendered on two 14-inch diagonal 
active matrix Liquid Crystal Displays (AMLCDs), each having an addressable resolution 
of 1024 pixels by 768 lines. Any type of instrument face or display can be rendered, 
including new and novel displays. 

The OTW scene is a photo-textured presentation rendered at a 50 degree horizontal by 
37.5 degree vertical field-of-view and displayed utilizing a video projection system at an 
addressable resolution of 1280 pixels by 1024 lines. The screen is positioned so that 
active display area subtends approximately 50 degrees horizontal to the pilot’s eye point. 
Both instrument displays and the OTW scene are rendered at a 30 Hz frame rate with a 
60 Hz display refresh rate. 

Two complete visual databases are available for experimental use. One visual database 
includes the Denver Stapelton Airport and the new Denver International Airport terminal 
areas. A second visual database includes the state of Virginia. The Virginia database 
contains six major airports at which takeoffs and landings can be made. Another 24 
airports are rendered at photographic quality to facilitate pilotage along several routes 
between NASA Langley, Newport News/Williamsburg, Blacksburg, Richmond, 
Manassas, Washington National and NASA Wallops Island runways. The environmental 
conditions can be varied to achieve any meteorological conditions required, i.e. overcast, 
low RVR, cloud decks, etc. 


4. Data Acquisition System Description 

The data acquisition system allows the pilot’s actions in the simulator to affect the state 
of the aero simulation model, and the outputs from the aero simulation model to be 
indicated in the cockpit displays and indicators. The data acquisition system is used to 
collect information about the pilot’s control inputs and switch actions, format the data, 
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and transfer the data to the simulation system for processing in the aero/engine/ 
environment models. 

Referring to Figures B-2 and B-3, a data acquisition controller system is hosted in a 
Pentium 60 - based PC. The data acquisition controller contains a microcontroller which 
performs all input / output (I/O) operations with the hardware in the simulator cab. 
Operations performed by the controller include: 

• Acquiring analog control position information 

• Performing the analog to digital (A/D) conversions on control position 
information 

• Acquiring switch position discreets 

• Driving indicators in the cab (e.g. Gear Position Indicators, Outer Marker , 
Middle Marker) 

• Acquiring frequency selections set in the COM, NAV 1, NAV 2, ADF, 
transponder and autopilot interfaces in the radio stack located in the cockpit 
center console 

• Acquiring the OBS 1 &2, DG and Baro Altimeter knob settings 

• Updating the frequency displays in the COM, NAV 1, NAV 2, and ADF 
radios 

Data is moved between the simulation models and the data acquisition controller via a 
process which runs in the Pentium 60 under the Window 95™ operating system. At a 
frequency of 30 Hz (7 Hz in the SGI), the simulation model sends a buffer of data over 
the Ethernet connection to the Pentium 60. The data acquisition process moves the buffer 
to the data acquisition controller card for processing. The data acquisition process then 
sends a buffer of the data most recently acquired from the cab to the simulation model at 
a rate of 50 Hz and awaits the next buffer of data from the simulation model. 

The modular nature of the data acquisition system allows additional modules to be added 
to drive additional LED displays and indicators or relays. Further, digital to analog 
modules are available for driving analog instrumentation. 


5. Simulation Control, Monitoring and Recording 

All aspects of the simulation are controlled by the simulation control and flight path 
monitoring process running in either the SGI Indigo or the PC with STAGE as shown in 
Figures B-2 and B-3. The simulation control process initializes the simulation models in 
the SGI Onyx or the PC with FLSim, performs real time data display, and presents a plan 
view of the aircraft’s position during operation of the simulation, similar to an ATC 
console. The system operator uses the simulation control to select various scenarios, 
position / reposition the aircraft model, and monitor scenario progress. 

Table B-2 lists the data dictionary of parameters available for collection and reduction. 
These items can be collected during the execution of any given scenario: 
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Table B-2. Real-time Parameters Displayed During Simulator Operations 


Parameter 

Parameter 

Parameter 

Parameter 

Airspeed (A/S) 

• Calculated A/S 

• Indicated A/S 

• True A/S 

Aerodynamic 

Coefficients 

* CL Total Lift 

* CD Total Drag 

9 CY Total Side Force 

* CM Total Pitching 
Moment 

* CR Total Rolling 
Moment 

* CN Total Yawing 
Moment 

Control Surface 
Deflection 

• Elevator 

• Rudder 

• Aileron 

• Aileron Trim 

• Rudder Trim 

• Trailing Edge 
Flaps 

• Left Spoiler 

• Right Spoiler 

Ground Contact 

Conditions 

(Landing) 

• Rate of Descent 

• Bank Angle 

• Side & Vertical 
Forces on Nose 
Gear 

• Side & Vertical 
Forces on Left 
Gear 

• Side & Vertical 
Forces on Right 
Gear 

Ground Speed 

Altitude 

• Pressure 

• AGL 

Position 

• Latitude 

• Longitude 

• Heading 


Aircraft Body 
Angles 

• Pitch 

• Roll 

• Yaw 

• Angle of Attack 

a 

• Side Slip p 

Atmospheric 

• OAT 

• Air Pressure 

• Baro Pressure 

Weight & Balance 

• Gross Weight 

• Payload 

• Total Fuel 

• CG relative to 
35% Mean 
Aerodynamic 
Chord (MAC) 


Z-Load (# Gs 
through the polar 
axis) 

Engine Thrust 

Rate of Climb 
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Table B-3. Data Dictionary of Recordable Parameters and Inducible Faults 


♦ 

Time in tenths of a second 

♦ 

Latitude 

♦ 

Longitude 

♦ 

Altitude 

♦ 

Airspeed 

♦ 

Heading 

♦ 

Aileron deflection 

♦ 

Elevator deflection 

♦ 

Rudder deflection 

♦ 

Throttle position 

♦ 

Prop position 

♦ 

Mixture position 

♦ 

Vertical Speed Indicator 

♦ 

Altitude above ground 

♦ 

Flight Path Marker X and Y positions 

♦ 

Flare symbol Y position 

♦ 

Database X and Y position of ownship 

♦ 

Left and Right Brake position 

Switch changes and positioning 

♦ 

Mags 

♦ 

Starter 

♦ 

Master 

♦ 

Pitot heat 

♦ 

Fuel pump 

♦ 

Nav lights 

♦ 

Flashing beacon 

♦ 

Landing lights 

♦ 

Taxi lights 

♦ 

Fuel selector 

♦ 

Toe brakes 

♦ 

Gear position 

♦ 

Flap position 

♦ 

Comm radio frequency 

♦ 

Nav 1 & 2 radio frequencies 

♦ 

Transponder frequency 

♦ 

ADF frequency 

♦ 

Autopilot switch positions 

♦ 

Any induced faults 

♦ 

Phase of flight 
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All of the above items are automatically sensed and recorded except for the phase of 
flight. This item is entered by point and click action on the I/O operator console. 

6. Simulation Models 

6.1 Aerodynamic Simulation Model 

The simulation model is a 6 Degree of Freedom (DOF) aerodynamic model that is: 

1) table driven in the SGI version and 

2) Formula driven in the PC version 

to provide the performance characteristics for the Piper Malibu 46P. The Piper Malibu 
currently in use represents a high performance single engine GA with a cruising speed of 
170 knots. The basic aero model can be adapted to any aircraft. 

The aerodynamic coefficients in the simulation model incorporate the non-linear 
characteristics of an operational airplane. These adjustments give the simulation model 
more realistic longitudinal handling characteristics and make it possible to flare the 
airplane to a maximum lift stall. 

6.2 Wind Models 

A cross wind model is available that can direct a cross wind over a large range from any 
direction. A 15 kt cross wind is shown relative to the nose of the aircraft in Figure B-13. 
Additional sophistication can be added to the wind model including gusts and wind from 
any direction. 



Figure B-13. Cross Wind Model 

A turbulence wind model is also available that applies a turbulence factor to the aircraft 
based on the level desired.cross wind over a large range from any direction. Four levels 
of turbulence are currently modeled ranging from mild to severe. These factors affect 
the aircrafts U, V, and W axis within the aero model. The effects of the turbulence are 
evident in the gauges and the out-the -window scene. Other windshear models are 
available on the PC version. 


119 



Appendix C. Description of Flight Information Services Data 
Link (FISDL) and Its Use from FAA 
Aeronautical Information Manual (AIM) 

7-1-11. Flight Information Services Data Link (FISDL) 

a. FISDL. Aeronautical weather and operational information may be displayed in 
the cockpit through the use of FISDL. FISDL systems are comprised of two basic 
types: broadcast systems and two-way systems. Broadcast system components 
include a terrestrial or pace-based transmitter, an aircraft receiver, and a cockpit 
display device. Two-way systems utilize transmitter/receivers at both the 
terrestrial or space-based site and the aircraft. 

1. Broadcast FISDL allows the pilot to passively collect weather and 
operational data and to call up that data for review at the appropriate time. 

In addition to text weather products, such as METAR's and TAF's, 
graphical weather products, such as radar composite/mosaic images may 
be provided to the cockpit. Two-way FISDL services permit the pilot to 
make specific weather and operational information requests for cockpit 
display. 

2. FISDL services are available from three types of service providers. 

(a) Through vendors operating under a service agreement with the 
FAA using broadcast data link on VHF aeronautical spectrum 
(products and services are defined under subparagraph c). 

(b) Through vendors operating under customer contract on 
aeronautical spectrum. 

(c) Through vendors operating under customer contract on other 
than aeronautical spectrum. 

3. FISDL is a method of disseminating aeronautical weather and 
operational data which augments pilot voice communication with Flight 
Service Stations (FSS's), other Air Traffic Control (ATC) facilities or 
Airline Operations Control Centers (AOCC's). FISDL does not replace 
pilot and controller/flight service specialist/aircraft dispatcher voice 
communication for critical weather or operational information 
interpretation. FISDL, however, can provide the background information 
which can abbreviate and greatly improve the usefulness of such 
communications. As such, FISDL serves to enhance pilot situational 
awareness and improve safety. 

b. Operational Use of FISDL. Regardless of the type of FISDL system being 
used, either under FAA service agreement or by an independent provider, several 
factors must be considered when using FISDL. 
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1. Before using FISDL in flight operations, pilots and other flight crew 
members should become completely familiar with the operation of the 
FISDL system to be used, airborne equipment to be used, including 
system architecture, airborne system components, service volume and 
other limitations of the particular system, modes of operation and the 
indications of various system failures. Users should also be familiar with 
the content and format of the services available from the FISDL 
provider(s). Sources of information which may provide this guidance 
include manufacturer's manuals, training programs and reference guides. 

2. FISDL does not serve as the sole source of aeronautical weather and 
operational information. ATC, FSS, and, if applicable, AOCC VHF/HF 
voice is the basic method of communicating aeronautical weather, special 
use airspace, NOT AM and other operational information to aircraft in 
flight. FISDL augments ATC/FSS/AOCC services, and, in some 
applications, offers the advantage of graphical data. By using FISDL for 
orientation, the usefulness of any information received from conventional 
voice sources may be greatly enhanced. FISDL may alert the pilot to 
specific areas of concern which will more accurately focus requests made 
to FSS or AOCC for inflight briefings or queries made to ATC. 

3. The aeronautical environment is constantly changing; often these 
changes occur quickly, and without warning. It is important that critical 
decisions be based on the most timely and appropriate data available. 
Consequently, when differences exist between FISDL and information 
obtained by voice communication with ATC, FSS, and/or AOCC (if 
applicable), pilots are cautioned to use the most recent data from the most 
authoritative source. 

4. FISDL products, such as ground-based radar precipitation maps, are not 
appropriate for use in tactical severe weather avoidance, such as 
negotiating a path through a weather hazard area (an area where a pilot 
cannot reliably divert around hazardous weather, such as a broken line of 
thunderstorms). FISDL supports strategic weather decision making such 
as route selection to avoid a weather hazard area in its entirety. The misuse 
of information beyond it's applicability may place the pilot and his/her 
aircraft in great jeopardy. In addition, FISDL should never be used in lieu 
of an individual pre-flight weather and flight planning briefing. 

5. FISDL supports better pilot decision making by increasing situational 
awareness. The best decision making is based on using information from a 
variety of sources. In addition to FISDL, pilots should take advantage of 
other weather/NAS status sources, including, but not limited to, Flight 
Service Stations, Flight Watch, other air traffic control facilities, airline 
operation control centers, pilot reports, and their own personal 
observations. 
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c. FAA FISDL. The FAA's FISDL system provides flight crews of properly 
equipped aircraft with a cockpit display of certain aeronautical weather and flight 
operational information. This information is displayed using both text and graphic 
format. This system is scheduled for initial operational capability (IOC) in the 
first quarter of calendar year 2000. The system is operated by vendors under a 
service agreement with the FAA, using broadcast data link on aeronautical 
spectrum on four 25 KHz spaced frequencies from 136.425 through 136.500 
MHz. FISDL is designed to provide coverage throughout the continental U.S. 
from 5,000 feet AGL to 17,500 feet MSL, except in those areas where this is 
unfeasible due to mountainous terrain. Aircraft operating near transmitter sites 
will receive useable FISDL signals at altitudes lower than 5000 feet AGL, 
including on the surface in some locations, depending on transmitter/aircraft line 
of sight geometry. Aircraft operating above 17,500 MSL may also receive useable 
FISDL signals under certain circumstances. 

1. FAA FISDL provides, free of charge, the following basic products: 

(a) Aviation Routine Weather Reports (METAR's). 

(b) Special Aviation Reports (SPECI's). 

(c) Terminal Area Forecasts (TAF's), and their amendments. 

(d) Significant Meteorological Information (SIGMET's). 

(e) Convective SIGMET's. 

(f) Airman's Meteorological Information (AIRMET's). 

(g) Pilot Reports (both urgent and routine) (PIREP's); and, 

(h) Severe Weather Forecast Alerts (AWW's) issued by the FAA 

or NWS. 

2 . The format and coding of these products are described in Advisory 
Circular AC-00-45, Aviation Weather Services, and paragraph 7-1-30. 

Key to Aerodrome Forecast (TAF) and Aviation Routine Weather Report 
(METAR). 

3 . Additional products, called Value-Added Products, are available from 
the vendors on a paid subscription basis. Details concerning the content, 
format, symbology and cost of these products may be obtained from the 
following vendors: 

(a) BENDIX/KING WxSIGHT 

Allied Signal, Inc. 

One Technology Center 

23500 West 105th Street 

Olathe, KS 66061 


122 


(913)712-2613 

www.bendixking.com 


(b) ARNAV Systems, Inc. 

16923 Meridian East 
P. 0. Box 73730 
Puyallup, WA 98373 
(253) 848-6060 
www.amav.com 

d. Non-FAA FISDL Systems. In addition to FAA FISDL, several commercial 
vendors provide customers with FISDF on both the aeronautical spectrum and 
other frequencies using a variety of data link protocols. In some cases, the 
vendors provide only the communications system which carries customer 
messages, such as the Aircraft Communications Addressing and Reporting 
System (ACARS) used by many air carrier and other operators. 

1. Operators using non-FAA FISDF for inflight weather and operational 
information should ensure that the products used conform to the 
FAA/NWS standards. Specifically, aviation weather information should 
meet the following criteria: 

(a) The products should be either FAA/NWS accepted aviation 
weather reports or products, or based on FAA/NWS accepted 
aviation weather reports or products. If products are used which do 
not meet this criteria, they should be so identified. The operator 
must determine the applicability of such products to flight 
operations. 

(b) In the case of a weather product which is the result of the 
application of a process which alters the form, function or content 
of the base FAA/NWS accepted weather product(s), that process, 
and any limitations to the application of the resultant product 
should be described in the vendor's user guidance material. 

2. An example would be a NEXRAD radar composite/mosaic map, which 
has been modified by changing the scaling resolution. The methodology of 
assigning reflectivity values to the resultant image components should be 
described in the vendor's guidance material to ensure that the user can 
accurately interpret the displayed data. 

3. To ensure airman compliance with Federal Aviation Regulations, 
National Airspace System (NAS) status products (such as NOTAM's, 
Special Use Airspace Status, etc.) and other government flight information 
should include verbatim transmissions of FAA products. If these products 
are modified, the modification process, and any limitations of the resultant 
product should be described in the vendor's user guidance. 
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Appendix D. NEXRAD Mosaic Images Used In Experiment 
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Appendix E. Description of National Convective Weather 
Forecast (NCWF) Product and Its Use from FAA Aeronautical 

Information Manual 


February 21, 2002 

7-1-14. National Convective Weather Forecast (NCWF) 

a. Description. 

1. The NCWF is an automatically generated depiction of: (1) current 
convection and (2) extrapolated significant current convection. It is a 
supplement to, but does NOT substitute for, the report and forecast 
information contained in Convective SIGMET's (see paragraph 7-l-6d ). 
Convection, particularly significant convection, is typically associated 
with thunderstorm activity. 

2. The National Weather Service Aviation Weather Center (AWC) updates 
the NCWF based on input from the Next Generation Weather Radar 
(NEXRAD) and cloud-to-ground lightning data. 

3. The NCWF is most accurate for long-lived mature multi-storm systems 
such as organized line storms. NCWF does not forecast initiation, growth 
or decay of thunderstorms. Therefore, NCWF tends to under-wam on new 
and growing storms and over-wam on dying storms. Forecast positions of 
small, isolated or weaker thunderstorms are not displayed. 

4. The NCWF area of coverage is limited to the 48 contiguous states. 

b. Attributes. 

1. The NCWF is updated frequently (every 5 minutes) using the most 
current available data. 

2. The NCWF is able to detect the existence of convective storm locations 
that agree very well with concurrent radar and lightning observations. 

3. The NCWF is a high-resolution forecast impacting a relatively small 
volume of airspace rather than covering large boxed areas. The location, 
speeds and directions of movement of multiple convective storms are 
depicted individually. 

4. The NCWF extrapolation forecasts are more accurate when predicting 
the location and size of well organized, unchanging convective storms 
moving at uniform speeds. The NCWF does not work well with sporadic, 
explosive cells developing and dissipating in minutes. 

5. In displaying forecast cell locations, the NCWF does NOT distinguish 
among level 3 through level 6 on the NCWF hazard scale (see TBL 7-1-3 ). 
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6. The NCWF may not detect or forecast: 

(a) Some embedded convection. 

(b) Low-topped convection containing little or no cloud-to-ground 
lightning (such as may occur in cool air masses). 

(c) Rapidly evolving convection. 

7. The NCWF cannot provide information on specific storm hazards such 
as hail, high winds or tornadoes. 

c. Availability and Use. 

1. The NCWF is available primarily via the Internet from the AWC 
Aviation Digital Data Service (ADDS) at 

http://adds.aviationweather.noaa.gov . Used in conjunction with other 
weather products such as Convective SIGMET's, the NCWF provides 
additional information for convective weather avoidance and flight 
planning. 

2. The NCWF access by Automated Flight Service Stations and their 
associated En Route Flight Advisory Service Facilities, Air Route Traffic 
Control Centers (ARTCC's) or Terminal Radar Approach Controls is 
planned but NOT currently available. 

NOTE- 

See AIM, paragraph 7-1-15, ATC Inflight Weather Avoidance Assistance, 
for further information. 

d. Display Summary. 

1. Existing convective hazards (based on NEXRAD and lightning data) 
are depicted using the color-coded 6-level NCWF hazard scale shown in 
TBL 7-1-3. In displaying forecast cell locations, the NCWF does NOT 
distinguish among level 3 through level 6. 


TBL 7-1-3 

NCWF Hazard Scale 


Level Color Effect 

5-6 

Red 

Thunderstorms may contain any or all of the following: 
severe turbulence, severe icing, hail, frequent lightning, 
tornadoes and low-level wind shear. The risk of hazardous 
weather generally increases with levels on the NCWF 
hazard scale. 

3-4 

Yellow 

Orange 

1-2 

Green 


NOTE- 

Although similar, the NCWF hazard scale levels are NOT identical to VIP 
levels. 


REFERENCE- 

Pilot/Controller Glossary Term- Radar Weather Echo Intensity Levels. 
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2. One-hour forecast locations of signification convection (NCWF hazard 
scale levels of 3 or greater) are depicted with blue polygons. Their 
directions of movement and storm tops are also shown. 

3. The Java display permits some degree of customization. Other means of 
viewing the NCWF may not offer these display options. Java display 
options include the following (see FIG 7-1-12 ): 

(a) "Current Convective Interest Grid." 

(b) "One-Hour Extrapolation Polygons." 

(c) "Previous Performance Polygons." 

(d) Storm speed and altitude of tops. 

(e) Overlays of: 

(1) Airport locations. 

(2) County boundaries. 

(3) ARTCC boundaries. 

(f) Aviation Routine Weather Reports (METAR's). 

(g) Unlimited customized zooms (by holding down the left mouse 
button and dragging to select the rectangle of coverage desired). 

4 . The JavaScript display options include the following (see FIG 7-1-13 ): 

(a) Current convective hazard "Detection" field. 

(b) 1 -hour extrapolation "Forecast" polygons. 

(c) Previous hour "Performance" polygons. 

(d) "2 hr Movie" loops of convective hazard detection fields (with 
forecast polygons included on the last frame). 

(e) "24 hr Movie" loops of convective hazard detection fields. 

(f) Zoomed views of: 

(1) ARTCC boundaries. 

(2) Certain major airports. 

(3) Seven geographical regions: Northwest, North Central, 
Northeast, Southwest, South Central, Southwest, the 48 
contiguous states. 

5. Additional information is available via the "FYI/Help" or "i" links on 
the Java and JavaScript displays, respectively. 
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Appendix F. NCWF Product Images Used In Experiment 
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Appendix G. Cockpit Weather Experiment III -- Subject Pilot 

Familiarization Briefings 


■ Control Pilot Group Briefing 

■ NEXRAD Image Looping Display Group 
Briefing 

■ NCWF Product Display Group Briefing 
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Control Pilot Group Familiarization Briefing 


Cockpit Display Experiment 
Briefing 
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Experiment Procedure 


Pre-Test Phase 


Test Phase 


Contact 
Potential 
Subjects & 
Set Datefs) 


Procedures 


Administer 

Knowledge 

“Quiz” 


Introductory 
Briefing and 
Simulator 
Orientation 



Structured Post-Flight 
Interview 

Confirm Observations & 
Investigate Pilot Decisions 



ERTI 



Today’s Flight Mission 
Situation 


•Three pilots have been friends since their military 
flying tours. 

•One of the friends lives in Newport News, VA, and 
will be flying his aircraft to Wallops Island for the 
wedding of one of the other friends. 

•The friend who will be flying the aircraft from 
Newport News - Williamsburg Airport has agreed 
to stop in Richmond enroute to Wallops Island to 
pick up the third friend (who is to be the best man) 
in Richmond. 


ERTI 
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Subject Pilot Schedule 


Time 

Activity 

0:20 

Introduction 

1:30 

Simulator Familiarization 

0:10 

Break 

0:30 

Flight Planning 

1:30 

Flight Experiment 

0:30 

Debriefing 


HRTI 



Today’s Mission 

(Continued) 

• The two friends who are traveling have planned 
their flight so as to attend a Friday night wedding 
rehearsal and dinner party. The rehearsal starts 
at 1700 hours local. 

•The pilot is delayed at work due to an important 
meeting, and the planned departure time from 
the Newport News airport has slipped to 1500 
hours local. 

•There is not sufficient time to drive from 
Newport News to Wallops and make the 
rehearsal. 


HRTI 

INTERNATIONAL 
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Simulation Hardware Configuration 


Closed Circuit 
TV (CCTV) and 


< Recorder 



Out The Window 
Scene 


Rapid Prototype Simulator Cab 



ISKH'TT 


Weather Information 
Facility 


p Simulation Cockpit Configuration 



HRTT 
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Simulation Cockpit Configuration 

Weather Information Sources 

• ATIS 

• Flight Service Station 

• Flight Watch 

• Virginia AWOS/ASOS Reports via radio 

• Air Traffic Control (IAW normal NAS procedures) 

Tower 

Departure 

Enroute 

Approach 


HRTI 



ERTI 



Simulation 

Cockpit 

Radio 

Stack 
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NEXRAD Image Looping Display Group 
Briefing 


Cockpit Display Experiment 
Briefing 


July 8 - August 24, 2002 



KJRTI 

I IN T r- R IN A T I O N A I. 




Experiment Procedure 


Pre-Test Phase 


Test Phase 


Contact 
Potential 
Subjects & 
Set Datefsi 


BRTI 


Administer 

Knowledge 

“Quiz” 


Procedures 


Introductory 
Briefing and 
Simulator 
Orientation 



Structured Post-Flight 
Interview 

Confirm Observations & 
Investigate Pilot Decisions 


Debrief 
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Subject Pilot Schedule 


Time 

Activity 

0:20 

Introduction 

1:30 

Simulator Familiarization 

0:10 

Break 

0:30 

Flight Planning 

1:30 

Flight Experiment 

0:30 

Debriefing 


HRTI 


Today’s Flight Mission 
Situation 

•Three pilots have been friends since their military 
flying tours. 

•One of the friends lives in Newport News, VA, and 
will be flying his aircraft to Wallops Island for the 
wedding of one of the other friends. 

•The friend who will be flying the aircraft from 
Newport News - Williamsburg Airport has agreed 
to stop in Richmond enroute to Wallops Island to 
pick up the third friend (who is to be the best man) 
in Richmond. 
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Today’s Mission 

(Continued) 

• The two friends who are traveling have planned 
their flight so as to attend a Friday night wedding 
rehearsal and dinner party. The rehearsal starts 
at 1700 hours local. 


•The pilot is delayed at work due to an important 
meeting, and the planned departure time from 
the Newport News airport has slipped to 1500 
hours local. 


•There is not sufficient time to drive from 
Newport News to Wallops and make the 
rehearsal. 


HRTI 



Simulation Hardware Configuration 


Closed Circuit 
TV (CCTV) and 
Recorder 



Out The Windo 
Scene 


Rapid Prototype Simulator Cab 



Weather Information 
Facility 


HRTI 
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Simulation Cockpit Configuration 


(a/s) @ 0 

Weather 

Display 

(v°k) K©@-'| 

(dG^ (vS^) 


®0©©J 


Engine 

Gauges 


Mags & 
Master 


) /!□! o ,Ji 


Electric Trim Switch 

Gear Switch 

Flap Switch 

Gear Indicators 
Throttle Quadrant 



Switches 

Taxi Lights 
Landing Lights 
Rotating Beacon 
NAV Lights 
Fuel Pump 
Pitot Heat 


Trim Indicator 


HRTI 



HRTI 



Simulation 

Cockpit 

Radio 

Stack 
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Simulation Cockpit Configuration 


Weather Information Sources 


ATIS 


• Flight Service Station 

• Flight Watch 

• Virginia AWOS/ASOS Reports via radio 

• Air Traffic Control (IAW normal NAS procedures) 

Tower 

Departure 

Enroute 

Approach 

•Data Link Cockpit Weather Information Display 


HRTI 


Cockpit Weather Display 
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Graphic Symbols 






w 




\ 



VOR 


\ 


Airport 


BRTI 

INTERNATIONAL 


Roads 


Coastline 



HRTI 

INTERNATIONAL 


Display Modes/Functions 


Display Mode: 
Graphics 
METAR 
NEX/METAR 


Movement Mode: 
Zoom 
Scroll 
Crosshair 



Bnghtnes^J 
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METAR Display 



jj Graphical METAR Codes 


CEILING 

Red - Less than 500 ft. 
Yellow -500 to 1000 ft. 
Green - 1000 to 3000 ft. 


Blue - More than 3000 ft. 



VISIBILITY 

Red - Less than 1 mi. 
Yellow - 1 to 3 mi. 

Green - 3 to 5 mi. 

Blue - More than 5 mi. 


HRTI 


Red - Low IFR 
Yellow -IFR 
Green - Marginal VFR 
Blue - Unlimited 
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METAR Selection 


HRTI 



METAR Text 



HRTI 
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NEXRAD Intensity Levels: 

Green 


Light 

Yellow 

a 



Red 


1 

r 



Severe 



156 



NCWF Display Group Familiarization Briefing 


Cockpit Display Experiment 
Briefing 

July 8 - August 24, 2002 

O HRTT 



Experiment Procedure 


Pre-Test Phase 


Test Phase 
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Subject Pilot Schedule 


Time 

Activity 

0:20 

Introduction 

1:30 

Simulator Familiarization 

0:10 

Break 

0:30 

Flight Planning 

1:30 

Flight Experiment 

0:30 

Debriefing 


HRTI 


Today’s Flight Mission 
Situation 

•Three pilots have been friends since their military 
flying tours. 

•One of the friends lives in Newport News, VA, and 
will be flying his aircraft to Wallops Island for the 
wedding of one of the other friends. 

•The friend who will be flying the aircraft from 
Newport News - Williamsburg Airport has agreed 
to stop in Richmond enroute to Wallops Island to 
pick up the third friend (who is to be the best man) 
in Richmond. 

mri 

INTERNATIONAL 
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Today’s Mission 

(Continued) 

• The two friends who are traveling have planned 
their flight so as to attend a Friday night wedding 
rehearsal and dinner party. The rehearsal starts 
at 1700 hours local. 


•The pilot is delayed at work due to an important 
meeting, and the planned departure time from 
the Newport News airport has slipped to 1500 
hours local. 


•There is not sufficient time to drive from 
Newport News to Wallops and make the 
rehearsal. 

HRTI 
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Simulation Hardware Configuration 


Closed Circuit 
TV (CCTV) and 
Recorder 



Out The Window 
Scene 


Rapid Prototype Simulator Cab 


ATC 

Controller 



Scenario 

Controller 



HRTI 
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Simulation Cockpit Configuration 


© 0 0 

Weather 

Display 

(wp) a©® 1 

@@@ 


® ©®© J 


Engine 

Gauges 


Mags & 
Master 



Electric Trim Switch 

Gear Switch 

Flap Switch 

Gear Indicators 
Throttle Quadrant 


Switches 

Taxi Lights 
Landing Lights 
Rotating Beacon 
NAV Lights 
Fuel Pump 
Pitot Heat 


Trim Indicator 


HRTI 



HRTI 



Simulation 

Cockpit 

Radio 

Stack 
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Simulation Cockpit Configuration 

Weather Information Sources 

• ATIS 


• Flight Service Station 

• Flight Watch 

• Virginia AWOS/ASOS Reports via radio 

• Air Traffic Control (IAW normal NAS procedures) 

Tower 

Departure 

Enroute 

Approach 

•Data Link Cockpit Weather Information Display 


HRTI 


Cockpit Weather Display 



HRTI 

INTERNATIONAL 
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Graphic Symbols 










X 



VOR 


X 


Airport 


BRXI Roads I 

,w TERNATIO Nil 


Coastline 



BRTI 
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Display Modes/Functions 


Display Mode: 
Graphics 
METAR 
NEX/METAR 


Movement Mode: 
Zoom 
Scroll 
Crosshair 



J ^nghtnes^ 
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METAR Display 




Graphical METAR Codes 


CEILING 

Red - Less than 500 ft. 
Yellow -500 to 1000 ft. 
Green - 1000 to 3000 ft. 
Blue - More than 3000 ft. 


Red -LowlFR 
Yellow -IFR 
Green - Marginal VFR 
Blue - Unlimited 


VISIBILITY 

Red - Less than 1 mi. 
Yellow - 1 to 3 mi. 

Green - 3 to 5 mi. 

Blue - More than 5 mi. 


K3RTI 
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METAR Selection 



METAR Text 



BRTT 
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NCWF Hazard Field 


HR.TI 
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DataTime: Jin 02. 1 99& 03:50 GMT Forecast Valid Time: Itm 02. 1 


GMT 


HRTI 


NCWF Final Product 


15 + 


6-14 



NCWF Summary 


Hazard Field 

Approximate Correlations 


VIL 

(kg/m 2 ) 

Ltg. 

Rate (/10 
min) 

Effect 

Airborne 

Radar 

Reflectivity 

(dBZ) 

VIP 

5-6 

12 + 

15 + 

Severe Turbulence, 
Lightning Hail likely 

Level 4 Very 
Hazardous 

50 + 

5-6 

4 

6.9 - 12.0 

6-14 

Possible Severe 
Turbulence, Lightning 

Level 3 
Hazardous 

45-49 

4 

3 

3.5 -6.9 

3-5 

Possible Severe 
Turbulence, Lightning 

Level 3 
Hazardous 

40-44 

3 

1-2 

0-3.5 

NA 

Light to Mod. Turbulence, 
Possible Lightning 

Level 1-2 
weak-moderate 

0-39 

1-2 


KRTI 
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NCWF Intensity Levels: 
Lt. Green □ Light 

Yellow □ | 

Red * Severe 


HRTI 



National Convective Weather Forecast 

(NCWF) 

• Based on NEXRAD and cloud-to-ground lightning data 

• Most accurate for long-lived mature multi-storm systems 

• Does not forecast initiation, growth or decay of 

thunderstorms 

• Tends to under- warn on new, growing storms and over- warn 

on dying storms 

• Does not display forecast positions of small, isolated or weaker 

thunderstorms 


• Forecast does not distinguish between level 3 through level 6 

on the NCWF hazard scale 

• May not detect or forecast: 

- some embedded convection, 

- low-topped convection containing little or no cloud-to- 

ground lightning (such as may occur in cool air masses), 


HRTI 


rapidly evolving convection 
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National Convective Weather Forecast 

(NCWF) 


• NCWF is currently available via the Internet from the 
Aviation Weather Center 


• Is to be used in conjunction with other weather products 
such as Convective Sigmets to provide additional information 
for flight planning 

• Not to be used as a substitute for Convective SIGMETs 


• Cannot provide information on specific storm hazards such 
as hail, high winds, or tornadoes 

• Access by Automated FSS, ARTCCs or Terminal Radar 
Approach Control planned but not currently available. 

mji 

INTERNATIONAL 
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Appendix H. Training & Proficiency Criteria 


In addition to a familiarization session of approximately 45 minutes in which the subject 
pilots were briefed on all functions of the simulation facility including the weather 
displays (for those subjects having a weather display), all pilots were provided training in 
the simulation facility itself so as to become sufficiently proficient in flying the simulator 
to accomplish the mission. The researcher/trainer guided the subject pilot through 
maneuvers to acquaint them with the operation and performance of the simulator. 
Additional instruction was given to the pilots on the weather display. 

Weather Display Experiment Simulator Familiarization Flight Syllabus 

1 . General explanation of cockpit layout: 

Primary flight instruments 

Secondary instruments 

Sub panel controls and systems 

Yoke controls 

Radios, Autopilot, Intercom 

Charts 

2. Checklist explanation 

3. Engine start and taxi 

4. Run-up and system check 

5. Normal takeoff and climb 

6. Level off at 2000 feet (± 100 feet) 

7. Shallow and steep banked turns to a heading (±10 degrees) 

8. Autopilot: 

Engage/disengage 
Pitch modifier 
Altitude hold 
Altitude modifier 
Heading hold 

9. VOR and ILS operation (with and without AP) 

10. Use of all functions of weather display (with and without AP) 

11. Vectors to normal VFR landing, touch-and-go (±10 kts) 

12. Go-around (±10 kts) 
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13. Vectors to IFR approach and landing 

14. Follow up approaches and landings if required until subject pilot able to accomplish 
all functions of mission without assistance from instructor 
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Appendix I. Observer Forms 


■ Richmond Leg 

■ Wallops Leg 
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Use of Wx Information Sources: 


Locations 


Decision Quality: 

1 = The pilot continued the approach into poor weather and was waved 
off the approach (by the tower controller) at the Final Approach Fix 
(outer marker). 

2 = The pilot abandoned the approach less than five (5) miles outside of 
the outer marker, but flew within five (5) miles of the red NEXRAD 
image cell, while in the Richmond area. 

3 = The pilot abandoned the approach by their own decision less than 
five (5) miles outside of the outer marker, and flew more than five (5) 
miles from a red NEXRAD image cell, while in the Richmond area. 

4 = The pilot abandoned the approach more than five (5) miles outside 
of the outermarker, and flew more than five (5) miles from a red 
NEXRAD image cell. 


ATIS 

ASOS 

AWOS 

ATC 

Departure Control 
Tower 

Approach Control 
Flight Watch 
_FSS 
METAR 
_Wx Display 


Workload: 


Extent to which the pilot relied on the autopilot to fly the airplane. 


None of the Time 


All of the Time 


Pilot’s Workload 


Extremely Low 


Extremely High 


Workload Considerations: 

Deviations from 

assigned flight path 

Procedural Errors 

Judgment Errors 

Communication 

Lapses/Errors 
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Observer Form Wallops 

Condition: Pilot #: Date: Time: Observer: 


Comments: 




Time Events 








































Use of Wx Information Sources: 

Types Locations Z-Times 

ATIS 
ASOS 
AWOS 
ATC 









Decision Quality: 

1 = The pilot flew within 10 miles of a red cell while circumventing the storms 
over the bay using the pilot’s own route planning. 

2 = The pilot flew within ten miles of a red cell, but only because of a delayed 
turn or distraction, but the intent was to circumvent by a at least 10 miles. 

3 = The pilot flew within 10 miles of a red cell, but was following vectors from 
ATC and for whatever reason, ATC vectored the pilot to within that 10 miles. 

4 = The pilot avoided a red cell by 10 miles or more. 

Departure Control 
Tower 

Approach Control 
Flight Watch 
FSS 

METAR 
Wx Display 


Extent to which the pilot relied on the autopilot to fly the airplane. 

i 1 

None of the Time All of the Time 

Pilot’s Workload 

1 1 1 

Workload Considerations: 

Deviations from 

assigned flight path 

Procedural Errors 

Judgment Errors 

Extremely Low Extremely High 

Communication 

Lapses/Errors 
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Appendix J. Immediate Reaction Questionnaires 

■ Control Group 

■ Weather Display Groups 

Immediate Reaction Questionnaire for Control Group 

Please use the following scales to rate how much you agree or disagree with the 
statements offered. If some items appear to overlap, do not be concerned, but attempt 
to answer each on its own terms. There are no “right” or “wrong” answers, nor any 
agenda of preferred responses being sought by the researchers. 

1 . 1 took the wedding scenario as a serious personal commitment, in the sense that I 
factored the time pressure into my decision-making. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 

2. 1 tried to systematically sample ah sources of weather information open to me. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 

3. 1 felt comfortable with the autopilot, in terms of understanding its use and operation. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 

4. Without the autopilot, my completion of the flight would have been compromised. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 

5. At the time of my arrival to the Richmond airport, I knew that there was a storm — 
(circle one) 

a. about 10 nm northwest of the airport 

b. about 5 nm northwest of the airport 

c. near the airport 

d. right at the airport 

6. At the time I was en-route to Wallops Island, I saw across my path of direct flight, what 
I took to be — 

a. a penetrable storm 

b. a navigable opening between convective cells 

c. a non-navigable opening between cells 

d. a wall of convective activity requiring diversion. 
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7. 1 felt that I had adequate sources of weather information to make confident decisions. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 
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IMMEDIATE REACTION QUESTIONNAIRE 

for 

NCWF and NEXRAD Looping Subject Pilot Groups 

Please use the following scales to rate how much you agree or disagree with the 
statements offered. If some items appear to overlap, do not be concerned, but attempt 
to answer each on its own terms. There are no “right” or “wrong” answers, nor any 
agenda of preferred responses being sought by the researchers. 


1 . 1 took the wedding scenario as a serious personal commitment, in the sense that I 
factored the time pressure into my decision-making. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 


2. An advantage of the onboard Weather Display was showing the graphical depiction of 
the weather to aid in my decision-making. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 


3. The weather display increased my situational awareness during the decision-making 
process. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 


4. 1 tried to systematically sample ah sources of weather information open to me. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 


5. 1 used the weather display, but felt the need to crosscheck or verify my conclusions from 
conventional weather data sources (ATC, etc.). 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 
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6. 1 felt comfortable with the autopilot, in terms of understanding its use and operation. 


Disagree 


Disagree No Opinion Agree Agree 

Somewhat Somewhat 


7. Without the autopilot, my completion of the flight would have been compromised. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 


8. The degree of validity and timeliness of the weather data appearing on the display was a 
factor I felt that I held in mind as I flew. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 


9. 1 have been monitoring the weather age display regularly in my instrument scan. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 


10. At the time of my arrival to the Richmond airport, I knew that there was a storm — 
(circle one) 

a. about 10 nm northwest of the airport 

b. about 5 nm northwest of the airport 

c. near the airport 

d. right at the airport 


1 1. At the time I was en-route to Wallops Island, I saw across my path of direct flight, 
what I took to be — 

a. a penetrable storm 

b. a navigable opening between convective cells 

c. a non-navigable opening between cells 

d. a wall of convective activity requiring diversion. 
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12. On the weather display, I found the positional accuracy of the aircraft icon to be 
adequate. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 


13. In using the weather display, I felt that I precisely knew the aircraft position relative to 
any storms. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 


14. 1 felt that I had adequate sources of weather information to make confident decisions. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 

15. The weather display increased head-down time and scanning workload 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 

16. I was aware of the latency of the weather data appearing on screen as I flew and 
assessed alternative flight paths. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 

17. I found the range rings very useful in enhancing my weather cell awareness. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 

18. The presentation of flight path waypoints facilitated my decision-making. 

Disagree Disagree No Opinion Agree Agree 

Somewhat Somewhat 
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Appendix K. Structured Interview Guide 


Richmond Decision Interview 

The following questions refer to only the leg between Newport News and Richmond. 


Decision Rationale 

1. What led you to make the decision to: 


2. What information did you use to make that decision? 


3. Do you feel that you had enough information to make a sound and confident judgement 
of the situation? 


4. What were your primary and secondary sources of weather information? 

Primary: 

Secondary: 


5. Was there any information that you lacked and would have liked to have? 


6a. Was there information that was available, but that you didn’t use? 


6b. Why didn’t you obtain that information? 
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7. Do you feel that you had enough time to gather all the information that you wanted? 


8. Did you ever consider holding? 


Wallops Decision Interview 

The following questions refer to only the leg between Richmond and Wallops Island. 


Decision Rationale 

9. What led you to make the decision to: 


10. What information did you use to make that decision? 


1 1. Do you feel that you had enough information to make a sound and confident judgement 
of the situation? 


12. What were your primary and secondary sources of weather information? 

Primary: 

Secondary: 


13. Was there any information that you lacked and would have liked to have? 
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14a. Was there information that was available, but that you didn’t use? 


14b. Why didn’t you obtain that information? 


15. Do you feel that you had enough time to gather all the information that you wanted? 


The following questions apply to the entire flight. 


Weather Interpretation 

17a. How close are you willing to fly near hazardous weather conditions, and what do you 
consider hazardous conditions? 


18. How close do you think that you flew to a red cell? 

Richmond — 

Wallops — 


Routing 

19. Did ATC help or hinder your route planning? 

20. At any time, did you consider totally aborting the mission? 

21. If you had a chance to do the flight again, would you do anything different? 
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Weather Display 


22. How would you compare the weather display information to other sources of weather 
information (ATIS, Flight Watch, etc.)? 


23. Did you use the weather display to help you navigate? 


24. Do you feel that the weather display increased or decreased your workload? 


25. Did you trust the weather display to give you correct information? 


26. Was there any confusion with the ownship symbology? 


27. How did you determine your distance to the weather cells? 


Comfort Zone 

28a. Did you feel ahead of the airplane? 


28b. If not, at what times did you feel behind the airplane, and what was the biggest 
contributor? 
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Appendix L. General Aviation Questionnaire 

(with Answers to Weather Knowledge Questions shaded) 

Thank you for participating in our Research Triangle Institute/NASA/FAA evaluation of 
advanced aviation technologies. We would like to learn a little more about your aviation 
knowledge before you participate in our study. Please take a couple of minutes to answer a 
few questions. Your answers are strictly confidential and will not be released. 

Name: Date: 

Phone number: E-Mail: 

1 . How many years have you been a pilot? 

2. What is your level of pilot certification (circle one)? 

Recreational Private Commercial Airline Transport 

3. What is your approximate number of total flight hours? 

4. Are you an instrument rated pilot? 

If so, are you current to fly instruments? 

5. What does a narrow temperature/dewpoint spread mean? 

Possible Fog 

6. How many feet are there in a statute mile? 

7. What does RVR stand for, and what does it mean? 

Runway Visual Range. Visibility down Specific Runway 

8. What COMM frequency can you use to contact Flight Watch? 122,2, 122,0 

9. Briefly, describe class C and class G airspace. 

Class C: 

Class G: 
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10. How much does 20 gallons of 100 LL fuel weigh? 


1 1 . What instrument indications would you notice, on take-off, if the static ports were 
blocked? 


12. What are the altitude limits of class A airspace, and what flight rules apply when flying 
in that airspace? 


13. If you are flying eastbound, and you have a tailwind, would you typically be north or 
south of a low-pressure zone? 

South 


14. On a surface analysis weather chart, what do closely spaced isobars mean? 

High Winds 

15. What are you likely to see on the instruments if a pitot tube becomes blocked during 
the enroute phase of flight? Describe each phase. 

Level (accelerating): 

Climb: 

Descent: 


16. In what weather products can you find icing information? 

PIREPs, SIGMETs. TAFs, AIRMETs. Area Forecasts. Prognosis Charts. Composite 
Moisture Charts. Wind Aloft Tables 

17. What do boundary layer air, and surface winds near the ground have in common? 

Both are slower than surrounding air, due to friction of the surface. 
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18. On a weather chart, what do the following symbols stand for? 



A. 


Occluded Front 



Stationary Front 


19. What type of information is found in an FDC NOTAM? 

Regulatory Notices. Charting Changes 

20. If a thunderstorm is identified as being severe, or giving an intense radar echo, what 
does the AIM say about how far you should avoid the storm? 

20 miles 

21. What do the following METAR/TAF weather codes stand for? 


RA = Rain 

SO = Squall 

BR = Mist 

FZ = Freezing 

FC = Tornado 

DZ = Drizzle 

SH = Showers 

FU = Smoke 

FG = Foe 

GR = Hail 


SN = Snow 

IC = Ice Crystals 

HZ = Haze 

TS = Thunderstorm 


22. What is a void time clearance? 
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23. On a radar summary chart, what does the notation “NA” mean? 


Not available 


24. During a night time IFR flight, what clues suggest airframe icing? 


25. Please translate the following METAR weather report: 

METAR KDCA 291554Z 26012G18KT 10SM SCT040 BKN100 

15/05 A2985 


Regan National, 29 th day, 1554 Zulu time, wind 260 degrees at 12 knots - gusting 
to 18 knots, visibility 10 statute miles, scattered clouds at 4000 feet, broken 
clouds at 10,000 feet, temperature 15 degrees Celsius, dewpoint 5 degrees 
Celsius, altimeter 29.85 inches of Mercury. 


Thank you for taking the time to complete our questionnaire, we appreciate your help. If 
we select you for our simulator study of advanced technologies, we will contact you by 
phone or E-mail. 
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Appendix M. Air Traffic Control Scripts 

Communication Exchanges Between Pilot and ATC 

The following is a typical communication exchange for the mission. Each pilot deviated 
from this typical exchange, some more than others, but only to the extent of clarifying 
radio calls, routing changes and exchanges to gather weather information. 


FIRST LEG 


SUMMARY: INSTRUMENT FLIGHT FROM NEWPORT NEWS/ 
WILLIAMSBURG (PHF) AIRPORT TO RICHMOND INTERNATIONAL (RIC) 
AIRPORT VIA DIRECT HOPEWELL V260 RICHMOND. SEVERE 
THUNDERSTORM APPROACHING RIC. PILOT TO DECIDE W H ETHER TO 
CONTINUE APPROACH AND ATTEMPT LANDING AT RIC, HOLD 
AWAITING WEATHER IMPROVEMENT, OR BY-PASS RIC AND REQUEST 
CLEARANCE TO WALLOPS, VA (WAL) FLIGHT FACILITY. 


N73Y: (Tunes 128.65 for ATIS) 

ATIS: THIS IS NEWPORT NEWS WILLIAMSBURG INTERNATIONAL TOWER 
INFORMATION BRAVO. 1800 ZULU MEASURED CEILING 1000 OVERCAST 
VISIBILITY 3 MILES. TEMPERATURE14 DEWPOINT 12 WIND 090 AT 10 
ALTIMETER 29.92. LANDING AND DEPARTING RUNWAY 7. ILS RUNWAY 7 
APPROACH IN USE. ADVISE YOU HAVE BRAVO. 


N73Y: Newport News clearance delivery, Malibu 2573Y ready for clearance. (121.65) 

ATC: MALIBU 2573 Y CLEARED TO RICHMOND VOR VIA DIRECT 
HOPEWELL V260 RICHMOND MAINTAIN 5000. SQUAWK 1424. 


N73Y: Roger, cleared to Richmond via direct Hopewell V260 Richmond maintain 5000. 


(Tunes 121.9) 


N73Y: Newport News ground control, N73Y ready to taxi, have information Bravo. 
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ATC: N73Y, GROUND CONTROL, TAXI STRAIGHT AHEAD THEN LEFT TO 
RUNWAY 7. WHEN READY FOR TAKEOFF, CONTACT TOWER ON 118.7. 


N73Y: Malibu 73Y, Roger. 


(Tunes 118.7) 


N73Y: Tower, N73Y ready for takeoff. 


ATC: N73Y MAINTAIN RUNWAY HEADING FOR RADAR VECTORS 
HOPEWELL MAINTAIN 2000, EXPECT CLEARANCE TO 5000 WITHIN 5 
MINUTES AFTER DEPARTURE. CLEARED FOR TAKEOFF RUNWAY 7. 


N73Y: Malibu 73Y Roger, cleared for takeoff. 
(Departs) 


ATC: MALIBU 73 Y CONTACT NORFOLK DEPARTURE CONTROL ON 124.9. 


(Tunes 124.9) 

N73Y: Norfolk departure control, this is N73Y climbing to 2000 on runway heading. 

ATC: N73Y ROGER, IN RADAR CONTACT. TURN LEFT PROCEED DIRECT 
HOPEWELL, CLIMB AND MAINTAIN 5000. 

N73Y: Malibu 73Y Roger, Proceeding direct Hopewell. 

N73Y: Norfolk departure control, request permission to leave frequency for Richmond 
ATIS. 
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ATC: N73Y FREQUENCY CHANGE APPROVED. ADVISE W H EN BACK ON 
MY FREQUENCY. 


(N73Y tunes 1 19.15 for RIC ATIS) 


ATIS: THIS IS RICHMOND TOWER INFORMATION DELTA. 1910 ZULU 
MEASURED CEILING 200 OVERCAST VISIBILITY THREE QUARTERS 
THUNDERSTORMS MODERATE RAIN SHOWERS TEMPERATURE 14 
DEWPOINT 12 WIND 300 AT 10 ALTIMETER 29.92. ILS RUNWAY 34 
APPROACH IN USE. LANDING AND DEPARTING ON RUNWAY 34. ADVISE 
YOU HAVE DELTA. 


(Tunes 124.9) 

N73Y: Departure control, N73Y back on your frequency. 


ATC: MALIBU 73 Y ROGER. 


ATC: MALIBU 73 Y CONTACT RICHMOND APPROACH CONTROL ON 134.7. 


(Tunes 134.7) 

N73Y: Richmond approach control, this is Malibu 73Y. Have information Delta. 


ATC: N73Y, RICHMOND APPROACH CONTROL, ROGER, DESCEND AND 
MAINTAIN 2000. EXPECT VECTORS TO ILS RUNWAY 34 APPROACH. 


N73Y: N73Y, Roger, descending to 2000. 


ATC: N73Y DEPART HOPEWELL VOR HEADING 300 FOR A VECTOR TO ILS 
RUNWAY 34 FINAL APPROACH COURSE. 
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N73Y: N73Y, Roger, depart Hopewell heading 300 for vector to ILS runway 34 approach 
course. 


ATC: MALIBU 73 Y, 4 MILES SOUTHEAST OF KAFKA, MAINTAIN 2000 UNTIL 
ESTABLISHED ON THE LOCALIZER, CLEARED FOR ILS RUNWAY 34 
APPROACH. CONTACT TOWER ON 121.1 PASSING KAFKA. 


N73Y: Malibu 73 Y, Roger, cleared for approach, tower 121.1 at KAFKA. 


(Tunes 121.1) 

ATC BROADCAST: ATTENTION ALL AIRCRAFT IN RICHMOND AREA. LOW 
LEVEL WINDSHEAR ADVISORIES IN EFFECT FOR RICHMOND 
INTERNATIONAL AIRPORT. 


(ATC TO IMPROVISE HOLDING, CLEARANCE TO WALLOPS, OR MISSED 
APPROACH DEPENDING ON PILOTS DECISION/REQUEST WITH WEATHER 
ENCOUNTERED.) 
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2ND LEG 


SUMMARY: INSTRUMENT FLIGHT FROM RICHMOND (RIC) TO WALLOPS 
FLIGHT FACILITY (WAL) VIA RICHMOND DIRECT HARCUM DIRECT 
JAMIE VI MAGGO. AT PILOT’S REQUEST AFTER HOLDING OR 
EXECUTING A MISSED APPROACH AT RICHMOND. 

ATC: MALIBU 73 Y CLEARED TO THE MAGGO INTERSECTION VIA DIRECT 
HARCUM DIRECT JAMIE VI MAGGO. CLIMB AND MAINTAIN 5000 
CONTACT RICHMOND DEPARTURE CONTROL ON 126.4. 


N73Y: Malibu 73Y, Roger, proceeding direct Harcum climbing to 5000, changing to 126.4. 
(Tunes 126.4) 

N73Y: Richmond departure control Malibu 73 Y proceeding direct Harcum climbing to 
5000. 

ATC: N73Y, ROGER, IN RADAR CONTACT. 

N73Y: Departure control, N73Y, request permission to leave frequency for Wallops ASOS 
information. 

ATC: N73Y FREQUENCY CHANGE APPROVED. ADVISE W H EN BACK ON 
MY FREQUENCY. 


(Tunes 119.175) 

WALLOPS AUTOMATED SURFACE OBSERVATION WIND 170 AT 6 VISIBILITY 
3 MILES. MEASURED CEILING 1000 BROKEN TEMPERATURE 23 DEWPOINT 
16 ALTIMETER 29.92. 


(Tunes 126.4) 

N73Y: Richmond departure control, Malibu 73 Y back on your frequency. 

ATC: MALIBU 73Y, ROGER. 

ATC: (When N73Y is approximately 10-15 nm from weather cells depicted.) N73Y, I 
SHOW WEATHER AHEAD. ADVISE INTENTIONS. 


(Possible requests from N73Y as weather is encountered.) 
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N73Y: 1) Request deviation to south/north to avoid weather. 

2) What do you show for weather on my route of fight? 

3) Request vector around weather. 

4) Request a new route/altitude to avoid weather. 

5) Request frequency change for Flight Watch or FSS. 


ATC: RESPOND TO SPECIFIC REQUEST, I. E: 

1) UNABLE TO APPROVE DEVIATION TO THE NORTH. RESTRICTED 
AREA 6609 IN USE. 

2) DEVIATION TO THE SOUTH APPROVED. 

3) I SHOW HEAVY WEATHER ON YOUR PROJECTED FLIGHT PATH. 

4) ROGER, TURN RIGHT HEADING FORA VECTOR SOUTH OF 

WEATHER. 

5) FREQUENCY CHANGE APPROVED. ADVISE WHEN BACK ON MY 

FREQUENCY. 


ATC: N73Y CLEAR OF WEATHER FLY HEADING FOR VECTOR TO VI. 


N73Y: N73Y, Roger, turning to headin g 

ATC: N73Y CONTACT PATUXENT APPROACH CONTROL ON 127.95. 


N73Y: N73Y Roger changing to 127.95 
(Tunes to 127.95) 

N73Y: Patuxent approach control, this is Malibu 73Y. 

ATC: N73Y THIS IS PATUXENT APPROACH CONTROL, EXPECT VOR/DME 
RUNWAY 10 APPROACH TO WALLOPS. ALTIMETER 29.92. 

N73Y: N73Y, Roger. 

ATC: N73Y, DESCEND AND MAINTAIN 2000. 

N73Y: N73Y, Roger, leaving 5000 for 2000. 

ATC: (5 Miles south ofMAGGO) TURN RIGHT HEADING 060 INTERCEPT THE 
SALISBURY 24.1 MILE ARC CLEARED FOR VOR/DME RUNWA Y 10 APPROACH. 

N73Y: N73Y, Roger, heading 060 to the arc, cleared for VOR/DME Runway 10 approach. 
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ATC: MALIBU 73Y CONTACT WALLOPS TOWER ON 126.5. 


N73Y: Malibu 73Y , Roger changing to tower. 

(Tunes 126.5) 

N73Y: Wallops Tower, this is Malibu 73 Y on approach to runway 10. 


ATC: MALIBU 73Y, WALLOPS TOWER, WIND 170 AT 6, CLEARED TO LAND 
RUNWAY 10 . 
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Appendix N. Enroute Weather Report Scripts 


The following weather report script was available to the Air Traffic Controller to be used 
as updated weather information. The reports were available while the mission was in 
progress, but the information was only given to the pilot if requested. These reports were 
available through the Flight Service Station radio, Enroute Flight Advisory Service (Flight 
Watch) and Air Traffic Control frequencies. Also included in this Appendix are the scripts 
used in the pre-recorded weather broadcasts of: Newport News ATIS, Richmond ATIS and 
Wallops Island ASOS. 

En-route Abbreviated Weather Reports 

AIRMET (WA) TANGO FOR OCNL MOD TURB BLO 060 for MD, VA and NC is 
current. 

ZDC CWA01 1855Z Valid Until 2100Z 
FROM CSN TO RIC TO DAN TO LYH TO CSN 

BKN AREA OF TSRA INCRG IN INTENSITY AND COVERAGE MOV EAST 

Washington Center Weather Advisory zero, one valid until two, one, zero, zero universal 
coordinated time. From Casanova, Virginia to Richmond, Virginia, to Danville, Virginia, 
to Lynchburg, Virginia, to Casanova, Virginia. Broken area of thunderstorms and rain 
increasing in intensity and coverage, moving east. 

ZDC CWA02 1855Z VALID UNTIL 2100Z 
FROM SBY225025 TO RIC090050 

BKN LINE OF TSRA INCRG IN INTENSITY AND COVERAGE MOV LITTLE 

Washington Center Weather Advisory zero, two valid until two, one, zero, zero universal 
coordinated time. From two, five miles Southwest of Salisbury, Maryland to Five, zero 
miles East of Richmond, Virginia. Broken line of thunderstorms and rain increasing in 
intensity and coverage, moving little. 

RIC SP 1910Z M002 OVC 3/4TRW 58/55/9012G16/992/TSTM OVHD 
OCNL LGTCCCG 

Richmond International Airport special weather report one, niner, one, zero universal 
coordinated time. Measured ceiling, two hundred, overcast, visibility _, thunderstorm, 
moderate rain showers, temperature five, eight, dew point five, five, wind zero, nine, zero, 
at one two, gusting one six, altimeter two niner nine two, thunderstorm overhead, 
occasional lightning cloud to cloud, and cloud to ground. 
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LKU SP 1905Z M005 OVC 1/2TRW+FG 57/57/2615G25/950/TSTM OVHD MOV E 
OCNL LGTCCCG 

Louisa County, Freeman Airport special weather report one, niner, zero, five universal 
coordinated time. Measured ceiling five hundred overcast, visibility one half, 
thunderstorm, heavy rain showers, fog, temperature five, seven, dew point five, seven, 
wind two, six, zero at one, five gusting two, five, altimeter two, niner, five, zero, 
thunderstorm overhead moving East, occasional lightning cloud to cloud, and cloud to 
ground. 

WAL SA 1846Z (Current) 

MVP SA 1846Z (Current) 

SBY SA 1845Z (Current) 

UA: /OV RIC /TM 1900Z /FL 010-SFC /TP C210 /TB SVR /RM LLWS FA 

Urgent pilot report over Richmond, Virginia at one, niner, zero, zero universal coordinated 
time. From one thousand feet to the surface, a Cessna two, one, zero reported severe 
turbulence and low-level wind shear on final approach. 

UA: /OV SBY /TM 1905Z /FL 060 /TP BE55 /TB NEG /RM MANY BLD-UPS OVR 
BAY SW 

Pilot report over Salisbury, Maryland at one, niner, zero, five universal coordinated time. 
At six thousand feet, a Beech five, five reported negative turbulence, and many build-ups 
over the bay Southwest. 

UA: /OV RIC090050 /TM 1900Z /FL080 /TP PA46 /TB NEG /RM BLD-UPS OVR 
BAYN 

Pilot report five, zero miles East of Richmond, Virginia at one, niner, zero, zero universal 
coordinated time. At eight thousand feet, a Piper four, six reported negative turbulence and 
build-ups over the bay North. 

Satellite Imagery indicates solid Build-Ups forming throughout Central Virginia. 

Weather Radar indicates solid light to moderate precipitation with increasing areas of 
Heavy precipitation developing throughout Central Virginia moving eastward into the 
Richmond (RIC), and Mecklenburg-Brunswick (A VC) areas. 
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Newport News ATIS 

this is Newport News-Williamsburg International tower information bravo, 
eighteen hundred zulu 
measured ceiling one thousand overcast 
visibility three miles 
temperature one-four 
dew point one-two 
wind, zero-niner-zero at one-zero 
altimeter, two-niner-niner-two 
landing and departing runway seven 
ILS runway seven approach in use 
advise you have bravo 


Richmond ATIS 


this is Richmond tower information delta, nineteen-ten zulu 

measured ceiling two-hundred, overcast 

visibility three-quarters 

thunderstorms, moderate rain showers 

temperature one-four 

dewpoint one-two 

wind, three-zero-zero at one-zero 

altimeter, two-niner-niner-two 

ILS runway three-four approach in use 

landing and departing runway three-four 

advise you have delta 


Wallops Island ASPS 

Wallops automated surface observation 

wind, one-seven-zero at six 

visibility, three miles 

measured ceiling one -thousand, broken 

temperature, two-three 

dewpoint, one-six 

altimeter, two-niner-niner-two 
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Appendix O. Subject Pilots’ Self Assessment of Workload 


TLX Measures 


Pilot 

# 

Condition 

Mental 

Physical 

Temporal 

Perform- 

ance 

Effort 

Frustra- 

tion 

Composite 

Workload 

TLX 

1 

Looping 

16 

4 

12 

14 

14 

12 

72 

2 

NCWF 

16 

9 

11 

19 

10 

13 

78 

3 

Control 

17 

7 

11 

16 

16 

11 

78 

4 

NCWF 

17 

5 

16 

10 

16 

15 

79 

5 

Control 

11 

16 

7 

10 

15 

6 

65 

6 

Looping 

16 

1 

3 

11 

13 

7 

51 

7 

Looping 

16 

8 

14 

17 

17 

16 

88 

8 

Control 

14 

6 

6 

11 

15 

18 

70 

9 

NCWF 

18 

8 

15 

16 

16 

16 

89 

10 

Looping 

16 

9 

18 

8 

18 

18 

87 

11 

Control 

19 

5 

14 

6 

18 

2 

64 

12 

NCWF 

18 

7 

10 

16 

16 

16 

83 

13 

Looping 

15 

4 

4 

13 

16 

11 

63 

14 

NCWF 

15 

12 

11 

14 

14 

9 

75 

15 

Control 

15 

7 

11 

13 

14 

14 

74 

16 

Control 

16 

4 

11 

7 

18 

7 

63 

17 

NCWF 

16 

16 

18 

16 

17 

9 

92 

18 

Looping 

8 

9 

8 

12 

8 

11 

56 

19 

NCWF 

17 

9 

16 

20 

17 

16 

95 

20 

Looping 

16 

10 

19 

7 

16 

14 

82 

21 

Control 

20 

16 

16 

18 

18 

12 

100 

22 

Control 

17 

6 

12 

16 

18 

17 

86 

23 

NCWF 

20 

14 

17 

17 

16 

15 

99 

24 

Looping 

20 

6 

17 

5 

20 

16 

84 

25 

Looping 

11 

6 

20 

15 

17 

13 

82 

26 

NCWF 

15 

3 

18 

3 

17 

12 

68 

27 

Control 

18 

15 

16 

4 

17 

12 

82 

28 

NCWF 

17 

10 

14 

12 

17 

13 

83 

29 

Looping 

17 

12 

17 

17 

17 

17 

97 

30 

Control 

20 

11 

16 

11 

16 

16 

90 

31 

Looping 

17 

5 

11 

2 

17 

14 

66 

32 

Control 

17 

11 

8 

9 

12 

9 

66 

33 

NCWF 

19 

7 

14 

11 

13 

14 

78 

34 

NCWF 

16 

15 

18 

19 

17 

14 

99 

35 

Control 

16 

2 

11 

14 

19 

3 

65 

36 

Looping 

16 

16 

17 

3 

18 

11 

81 

37 

NCWF 

20 

4 

9 

4 

20 

2 

59 

38 

Looping 

16 

13 

15 

18 

12 

15 

89 
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TLX Measures 

Composite 


Pilot 

# 

Condition 

Mental 

Physical 

Temporal 

Perform- 

ance 

Effort 

Frustra- 

tion 

Workload 

TLX 

39 

Control 

17 

8 

15 

11 

14 

14 

79 

40 

NCWF 

18 

16 

15 

6 

16 

6 

77 

41 

Looping 

11 

3 

14 

5 

14 

15 

62 

42 

Control 

19 

14 

13 

12 

13 

7 

78 

43 

Control 

14 

15 

16 

1 

7 

11 

64 

44 

NCWF 

15 

11 

12 

16 

12 

7 

73 

45 

Looping 

15 

5 

5 

15 

18 

11 

69 

46 

Control 

17 

14 

17 

17 

16 

15 

96 

47 

NCWF 

18 

8 

18 

16 

17 

16 

93 

48 

Looping 

13 

4 

4 

11 

15 

7 

54 
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Appendix P. Subject Pilot Schedule And Experiment Group 

Assignments 
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Pilot # 

Cond. Name 

Schedule 

Run 

1 

N EXRAD 

7/8/2002 

0800 

2 

NCWF 

7/8/2002 

1300 

3 

Control 

7/9/2002 

0800 

4 

NCWF 

7/9/2002 

1300 

5 

Control 

7/12/2002 

1300 

6 

N EXRAD 

7/10/2002 

1300 

7 

N EXRAD 

7/11/2002 

0800 

8 

Control 

7/11/2002 

1300 

9 

NCWF 

7/12/2002 

0800 

10 

N EXRAD 

7/15/2002 

0800 

11 

Control 

7/15/2002 

1300 

12 

NCWF 

7/16/2002 

0800 

13 

N EXRAD 

7/16/2002 

1300 

14 

NCWF 

7/17/2002 

0800 

15 

Control 

7/17/2002 

1300 

16 

Control 

7/18/2002 

0800 

17 

NCWF 

7/19/2002 

0800 

18 

N EXRAD 

7/23/2002 

0800 

19 

NCWF 

7/23/2002 

1300 

20 

N EXRAD 

7/24/2002 

0800 

21 

Control 

7/24/2002 

1300 

22 

Control 

7/26/2002 

0800 

23 

NCWF 

7/26/2002 

1300 

24 

N EXRAD 

8/5/2002 

0800 

25 

N EXRAD 

8/5/2002 

1300 

26 

NCWF 

8/6/2002 

0800 

27 

Control 

8/6/2002 

1300 

28 

NCWF 

8/7/2002 

0800 

29 

N EXRAD 

8/7/2002 

1300 

30 

Control 

8/8/2002 

1300 

31 

N EXRAD 

8/9/2002 

0800 

32 

Control 

8/12/2002 

0800 

33 

NCWF 

8/12/2002 

1300 

34 

NCWF 

8/13/2002 

0800 

35 

Control 

8/13/2002 

1300 

36 

N EXRAD 

8/14/2002 

0800 

37 

NCWF 

8/14/2002 

1300 

38 

N EXRAD 

8/16/2002 

1300 

39 

Control 

8/19/2002 

1300 

40 

NCWF 

8/20/2002 

0800 

41 

N EXRAD 

8/21/2002 

0800 

42 

Control 

8/21/2002 

1300 

43 

Control 

8/22/2002 

0800 

44 

NCWF 

8/22/2002 

1300 

45 

N EXRAD 

8/23/2002 

0800 

46 

Control 

8/23/2002 

1300 

47 

NCWF 

8/24/2002 

0800 

48 

N EXRAD 

8/24/2002 

1300 
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Appendix Q. Flight Paths of Each Subject Pilot 



Subject 01 
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Subject 03 



Subject 04 


203 




Subject 05 



204 



37.7 



CCV 


37.6 


37.5 


37.4 


37.3 


37.2 


37.1 


37 


36.9 


36.8 


- 76.8 


- 76.6 


- 76.4 


- 76.2 


-76 


- 75.8 


Subject 07 



Subject 08 
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Subject 09 



Subject 10 


206 




Subject 11 



Subject 12 
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Subject 13 



Subject 14 


208 




Subject 15 



209 





Subject 18 


210 




- 77.4 - 77.2 -77 - 76.8 - 76.6 - 76.4 - 76.2 

Subject 19 


37.5 

mm 

ccv 

37.3 

■ 3^1 


137.1 1 


K7 


36.9 


-76 - 75.8 
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Subject 21 



212 




Subject 23 



Subject 24 
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214 




Subject 27 



215 




Subject 29 



Subject 30 


216 




Subject 31 


37.6 



- 77.4 - 77.2 -77 - 76.8 - 76.6 - 76.4 - 76.2 -76 - 75.8 


Subject 32 


217 





Subject 34 
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Subject 35 



Subject 36 
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Subject 37 



Subject 38 


220 




Subject 39 



Subject 40 
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Subject 41 



Subject 42 
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Subject 43 
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- 77.4 - 77.2 -77 - 76.8 - 76.6 - 76.4 - 76.2 

Subject 45 


37.5 

Wm 

ccv 

37.3 

■ 3^1 


137.1 1 


K7 


36.9 


-76 - 75.8 
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